I would like to search a large corpus for example sentences which contain exclusively or a high proportion of words from the N most common words in some frequency dictionary (the language of the corpus is not important right now).

Is there a standard method?

  • 1
    I'm voting to close this question as off-topic because it's about programming; no knowledge of linguistics is required to answer it.
    – Keelan
    Jan 31 '20 at 17:13
  • 2
    wortschatz.uni-leipzig.de is a good resource for your purposes, but you will have to iterate over the list of words of interest yourself. Jan 31 '20 at 17:31
  • 1
    @Keelan the question is asking whether it's a "standard method" (I'd say rather a "standard metric") to do this, and I assume they meant linguistically. They aren't asking how to achieve it by programming, they pretty much fully described how. So while I'm not sure there is a "standard method" for this particular thing, which is to say I strongly suspect there isn't, it's at least about linguistics to me.
    – LjL
    Feb 4 '20 at 18:09
  • 1
    @Keelan I don't believe one is meant to stop at the title... that's a very brief summary of what they want to ultimately achieve, but the details of what "easy" means and how they want to achieve it are in the post. Seems standard practice to me, but you're welcome to edit it as far as I'm concerned if you can find better wording.
    – LjL
    Feb 5 '20 at 13:17
  • 3
    I’m voting to close this question because it's more about programming than linguistics.
    – curiousdannii
    Apr 21 '20 at 22:03

Ranking sentences by the commonness of their constituent words is not a feature I've seen in corpus search tools like CQPweb or AntConc. You'll probably have to get your hands on a copy of the corpus data and write the code yourself to rank sentences this way.


Sounds about right for a v1, depending on how you define difficulty.

I would probably do the opposite: count how many words are not frequent.

And I would not make it a hard cutoff, but rather weight by rarity.

The challenge is dealing with tokens like prices or emails that won't be in your list (OOV).

And there is also the sequence level. Very long sentences of easy words are probably not easy.

For a working solution, you can try ModelFront. It's for translation risk, which does tend to correlate strongly with sentence difficulty, modulo length.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.