13

I'd like to download a frequency list of English words that are most common in newspapers, books and magazines. It should contain at least 50.000 words.

Is it possible to download such a list from the web? Could you help me start to have better overview where to search for frequency lists and orient better in this area?

There are some requirements: For example, I'd like the list not to contain junk words, like names of certain celebrities. Furthermore, it should also contain phrasal verbs like stand up or look for, correctly sorted.

Improve this question
2

3 Answers 3

Reset to default
6

You can download the datasets used by the Google Ngram Viewer. These -datasets are compiled from the Google's collection of scanned books (so, magazines and newspapers are not included, AFAIK). If you are looking for frequencies of individual words only (and not, say, pairs, triples etc), you can download only the 1-gram datasets.

In addition, the word frequencies are annotated by year:

As an example, here are the 30,000,000th and 30,000,001st lines from file 0 of the English 1-grams (googlebooks-eng-all-1gram-20090715-0.csv.zip):

circumvallate   1978   313    215   85
circumvallate   1979   183    147   77

The first line tells us that in 1978, the word "circumvallate" (which means "surround with a rampart or other fortification", in case you were wondering) occurred 313 times overall, on 215 distinct pages and in 85 distinct books from our sample.

Since these datasets are is licensed under a Creative Commons Attribution 3.0 Unported License, you are free to share, remix and make commercial use of them, as long as you obey the conditions specified in the license.

Improve this answer
4
  • 1
    Thank you, that's good, but aren't there included some junk words too? Like words of certain celebrities and popular topics of the year?
    – xralf
    Jun 17, 2012 at 16:58
  • Other thing is that a lot of words are two words like "stand up", "look up". Are these words correctly sorted according to frequencies?
    – xralf
    Jun 17, 2012 at 17:00
  • 1
    @xralf Add those points to your original question. Add also any detail in order to scope this question enough, thanks.
    – Alenanno
    Jun 17, 2012 at 22:34
  • For reasons that need not be discussed, I searched for "trepidate" from 1800-1814 and found a number of references that are to magazines.
    – Ross Millikan
    Jun 18, 2012 at 4:28
3

This question hasn't been updated in a while. Google also released a 1 Billion Word Benchmark Corpus: https://github.com/tensorflow/models/tree/master/research/lm_1b. It has a vocabulary size of about 800k, as reported on the repo. The corpus was collected from English language newswire services from all over the world.

I've processed the unigram frequency of the corpus, available on github here: https://github.com/langcomp/lm_1b/blob/master/evaluation_code/lm1b-1gram.tsv. I'm not sure why Google says a vocab of 800k, because my unigram frequency has about 2.4 million tokens.

Improve this answer
3

Expanding on Otavio Macedo's answer, if you just want plain txt files containing English words sorted by frequency without having to analyze Google Ngram Viewer's dataset yourself, Natural Language Corpus Data (companion material for the book Beautiful Data by Segaran and Hammerbacher) provides two useful files:

  • count_1w.txt — a file containing 1/3 million (333,333) of the most common English words ordered by frequency.
  • count_2w.txt — a file containing 1/4 million (250,000) of the most common two-word bigrams ordered by frequency.

The data from these lists are also from Google Ngram Viewer's dataset.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.