Extensive English word corpora

Question

I am looking for an extensive list of english words[including American and British... just an exhaustive list]. This list of english words should contain an exhaustive collection of all forms of all words. And I am looking for just the english words not the sentences.

I have checked out a few sources but could find complete list of words. I can see the root word but not the derived or forms of that that word.

I have mentioned examples:

Currently i have checked out:

1. nltk.corpus.word.words() -------------couldn't find word "vendors"[found "vendor"]
2. gcide : http://gcide.gnu.org.ua/ -----couldn't find word "vendors"[found "vendor"]
3. BNC http://www.natcorp.ox.ac.uk/ ------Its a collection of sentences broked into forms. I'll have to parse and collect words. Still that doesn't guarantees a collection of exhaustive words.
4. COCA https://www.english-corpora.org/coca/ -----couldn't even understand the data. But it sure wasn't it. I am looking further into it.

Any sort of help is much appreciated.

Currently I am using...

s0 = set(corrected_sp_ch_wrd)-set( list([ a.lower() for a in nltk.corpus.words.words()]) +\ list([ a.lower() for a in nltk.corpus.brown.words()]) +\ list([ a.lower() for a in nltk.corpus.gutenberg.words()]) +\ list([ a.lower() for a in nltk.corpus.webtext.words()]) +\ list([ a.lower() for a in nltk.corpus.gazetteers.words()]) +\ list([ a.lower() for a in nltk.corpus.inaugural.words()]) +\ list([ a.lower() for a in nltk.corpus.abc.words()]) +\ combined_global_list )

combined_global_list is collection of all external corpora i parsed.

But this is not enough. There are words i am missing.

Answer 1

When you are interested just in word forms, the Leipzig Corpora Collection is a good place to look for corpora. They have a lot of English corpora separated by geographic origin (not only British and American, but also other regional varieties, e.g., South African).

Answer 2

You're mixing two concepts: words found in dictionaries (like gcide), and words found in a corpus (like COCA). The words in the dictionary will be stored in their base/lemma form, hence, you won't always see words variations (past tense, plural form, etc.)

If you're just looking for words regardless of form, you'll want to read a large corpus and save the tokens. See: https://www.reddit.com/r/LanguageTechnology/comments/a50q9y/open_source_word_lists/ebj3qwz/?context=3

Answer 3

To elicit a more comprehensive answer, the question must first define what kinds of words are acceptable, and what are not. For example, most people reading this answer might recognize the word "tokenize", but you will not find the word in common dictionaries. The other approach is to tokenize a sufficiently comprehensive corpus (like the Wikipedia dump), but you will also end up extracting nonsensical words (typos, spam, etc.) and foreign words. The other problem with tokenizing a large corpus is that you may end up missing morphological variants of rarely used words. For example "zymurgy's" is accepted by a typical spell-checker, and is understood by anyone who understands what zymurgy means, but has a good chance of not being found in a large corpus.

If you are only concerned with getting a list of all variants of words found in a typical dictionary, you can generate it on your own using a spellchecker.

For example, hunspell (man 5 hunspell) tells us that "vendor/MS" allows for "vendor", "vendor's", and "vendors", and that "zoological/Y" allows for "zoological" and "zoologically". You can figure out the list of root words in hunspell's en_US-large.dic and derive all their morphological variants with the help of en_US-large.aff. Repeat the exercise with en_GB. The .dic and .aff files are simple enough to be handled in Python without a lot of effort.