2

I am looking for an extensive list of english words[including American and British... just an exhaustive list]. This list of english words should contain an exhaustive collection of all forms of all words. And I am looking for just the english words not the sentences.

I have checked out a few sources but could find complete list of words. I can see the root word but not the derived or forms of that that word.

I have mentioned examples:

Currently i have checked out:

  1. nltk.corpus.word.words() -------------couldn't find word "vendors"[found "vendor"]
  2. gcide : http://gcide.gnu.org.ua/ -----couldn't find word "vendors"[found "vendor"]
  3. BNC http://www.natcorp.ox.ac.uk/ ------Its a collection of sentences broked into forms. I'll have to parse and collect words. Still that doesn't guarantees a collection of exhaustive words.
  4. COCA https://www.english-corpora.org/coca/ -----couldn't even understand the data. But it sure wasn't it. I am looking further into it.

Any sort of help is much appreciated.

Currently I am using...

s0 = set(corrected_sp_ch_wrd)-set( list([ a.lower() for a in nltk.corpus.words.words()]) +\ list([ a.lower() for a in nltk.corpus.brown.words()]) +\ list([ a.lower() for a in nltk.corpus.gutenberg.words()]) +\ list([ a.lower() for a in nltk.corpus.webtext.words()]) +\ list([ a.lower() for a in nltk.corpus.gazetteers.words()]) +\ list([ a.lower() for a in nltk.corpus.inaugural.words()]) +\ list([ a.lower() for a in nltk.corpus.abc.words()]) +\ combined_global_list )

combined_global_list is collection of all external corpora i parsed.

But this is not enough. There are words i am missing.

  • 1
    What exactly is the goal? Corpora are famously not an ideal source for ‘exhaustive’ word lists (en.m.wikipedia.org/wiki/Heaps'_law), which is one reason specific wordlists exist: dictionaries. – Jeremy Needle Mar 26 at 17:40
  • I think "exhaustive" was supposed to mean "indexing all word-forms from the corpus" – vectory Mar 26 at 18:47
  • @JeremyNeedle i was desperate to find an exhaustive list of english words that contains all the words and all forms of words. Since gcide wasn't enough, I went with standard corpus to tokenize them and get list of words thinking there wont be any spelling mistakes. – Krishna Chouhan Mar 27 at 11:27
  • The union each {a.lower() for a in nltk.corpus.[...].words()} that you are using(words, brown, gutenberg, webtext, gazetteers, inaugural, abc) give a total of 288,556 words. OED(1989) has 171,476 words entries in current use. In what way do you feel what you already have is not enough? And most critically, how would you define an "English word?" How would you know if or not you have a "list of English words that contains all the words and all the forms of words?" – Taegyung Mar 27 at 13:54
  • I had another extensive list. I did set difference and few words i got which were not already present... ['transnationals', 'treeing', 'valuables', 'outlasts'...]. I guess there is not way to be sure that the list is exhaustive or not. Atleast I have no idea on how to get its exhaustiveness confirmed. – Krishna Chouhan Mar 27 at 15:40
1

When you are interested just in word forms, the Leipzig Corpora Collection is a good place to look for corpora. They have a lot of English corpora separated by geographic origin (not only British and American, but also other regional varieties, e.g., South African).

  • I tried using the eng_news_2016_1M-words.txt but i got words like: ['insurace', 'benfits', 'cheech'...]. Although there were valid words as well but no way that i could extract them out. – Krishna Chouhan Mar 27 at 16:50
  • "A huge amount of words" and "free from errors" are contradicting requirements, and you have to live with some trade-off between the two. When you have an error model (e.g., what are typical typos) you can check the word forms from the corpus against a smaller dictionary and sort out potential errors while retaining rare words. – jknappen Mar 27 at 17:12
0

You're mixing two concepts: words found in dictionaries (like gcide), and words found in a corpus (like COCA). The words in the dictionary will be stored in their base/lemma form, hence, you won't always see words variations (past tense, plural form, etc.)

If you're just looking for words regardless of form, you'll want to read a large corpus and save the tokens. See: https://www.reddit.com/r/LanguageTechnology/comments/a50q9y/open_source_word_lists/ebj3qwz/?context=3

  • i was desperate to find an exhaustive list of english words that contains all the words and all forms of words. Since gcide wasn't enough, I went with standard corpus to tokenize them and get list of words thinking there wont be any spell mistakes. To avoid spelling mistakes i didnt tokenize Wiki dump. @prash has mentioned this problem. – Krishna Chouhan Mar 27 at 11:25
0

To elicit a more comprehensive answer, the question must first define what kinds of words are acceptable, and what are not. For example, most people reading this answer might recognize the word "tokenize", but you will not find the word in common dictionaries. The other approach is to tokenize a sufficiently comprehensive corpus (like the Wikipedia dump), but you will also end up extracting nonsensical words (typos, spam, etc.) and foreign words. The other problem with tokenizing a large corpus is that you may end up missing morphological variants of rarely used words. For example "zymurgy's" is accepted by a typical spell-checker, and is understood by anyone who understands what zymurgy means, but has a good chance of not being found in a large corpus.

If you are only concerned with getting a list of all variants of words found in a typical dictionary, you can generate it on your own using a spellchecker.

For example, hunspell (man 5 hunspell) tells us that "vendor/MS" allows for "vendor", "vendor's", and "vendors", and that "zoological/Y" allows for "zoological" and "zoologically". You can figure out the list of root words in hunspell's en_US-large.dic and derive all their morphological variants with the help of en_US-large.aff. Repeat the exercise with en_GB. The .dic and .aff files are simple enough to be handled in Python without a lot of effort.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.