I'm very new to NLP and I'm sure this question is a rookie one. In NLP for English language, how do we tell if a word is commonly used, from one that is not commonly used? What is the simplest measure? e.g., in a NLP application for children's bedtime story the need is to avoid uncommon words.

Many thanks. BTW, this is my first question in this forum.

  • Simply investigate a corpus (i.e. a large collection of text) and extract the word frequencies. The higher the frequency, the more often it is used. – lemontree Jul 18 '16 at 18:20
  • 1
    Of course, you then need to be sure what you mean by a word, i.e. lemmas (word = its underlying "dictionary form") or inflected word forms like child - children, go - goes, counted as separate words. You're probably more interested in lemmas. – lemontree Jul 18 '16 at 18:22
  • A good, since easy-to-use tool to start with is Python's NLTK library, which also comes with a book providing explanations and exapmles. – lemontree Jul 18 '16 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.