I would like to search a large corpus for example sentences which contain exclusively or a high proportion of words from the N most common words in some frequency dictionary (the language of the corpus is not important right now).

Is there a standard method?

  • 1
    I'm voting to close this question as off-topic because it's about programming; no knowledge of linguistics is required to answer it.
    – Keelan
    Commented Jan 31, 2020 at 17:13
  • 2
    wortschatz.uni-leipzig.de is a good resource for your purposes, but you will have to iterate over the list of words of interest yourself. Commented Jan 31, 2020 at 17:31
  • 1
    @Keelan the question is asking whether it's a "standard method" (I'd say rather a "standard metric") to do this, and I assume they meant linguistically. They aren't asking how to achieve it by programming, they pretty much fully described how. So while I'm not sure there is a "standard method" for this particular thing, which is to say I strongly suspect there isn't, it's at least about linguistics to me.
    – LjL
    Commented Feb 4, 2020 at 18:09
  • 1
    @Keelan I don't believe one is meant to stop at the title... that's a very brief summary of what they want to ultimately achieve, but the details of what "easy" means and how they want to achieve it are in the post. Seems standard practice to me, but you're welcome to edit it as far as I'm concerned if you can find better wording.
    – LjL
    Commented Feb 5, 2020 at 13:17
  • 3
    I’m voting to close this question because it's more about programming than linguistics.
    – curiousdannii
    Commented Apr 21, 2020 at 22:03

2 Answers 2


Ranking sentences by the commonness of their constituent words is not a feature I've seen in corpus search tools like CQPweb or AntConc. You'll probably have to get your hands on a copy of the corpus data and write the code yourself to rank sentences this way.


Sounds about right for a v1, depending on how you define difficulty.

I would probably do the opposite: count how many words are not frequent.

And I would not make it a hard cutoff, but rather weight by rarity.

The challenge is dealing with tokens like prices or emails that won't be in your list (OOV).

And there is also the sequence level. Very long sentences of easy words are probably not easy.

For a working solution, you can try ModelFront. It's for translation risk, which does tend to correlate strongly with sentence difficulty, modulo length.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.