Take the 2-minute tour ×
Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It's 100% free, no registration required.

I'd like to download a frequency list of english words that are most common in newspapers, books and magazines. It should contain at least 50 000 words.

Is it possible to download such a list from the web? Could you help me start to have better overview where to search for frequency lists and orient better in this area?

There are some requirements: For example, I'd like the list not to contain junk words, like names of certain celebrities. Furthermore, it should also contain phrasal verbs like stand up or look for, correctly sorted.

share|improve this question
1  
I think your additional requirements here are just the kind of thing we're looking for to decide between what are good and bad resource requests for corpora. –  hippietrail Jun 18 '12 at 8:30
add comment

1 Answer

You can download the datasets used by the Google Ngram Viewer. These -datasets are compiled from the Google's collection of scanned books (so, magazines and newspapers are not included, AFAIK). If you are looking for frequencies of individual words only (and not, say, pairs, triples etc), you can download only the 1-gram datasets.

In addition, the word frequencies are annotated by year:

As an example, here are the 30,000,000th and 30,000,001st lines from file 0 of the English 1-grams (googlebooks-eng-all-1gram-20090715-0.csv.zip):

circumvallate   1978   313    215   85
circumvallate   1979   183    147   77

The first line tells us that in 1978, the word "circumvallate" (which means "surround with a rampart or other fortification", in case you were wondering) occurred 313 times overall, on 215 distinct pages and in 85 distinct books from our sample.

Since these datasets are is licensed under a Creative Commons Attribution 3.0 Unported License, you are free to share, remix and make commercial use of them, as long as you obey the conditions specified in the license.

share|improve this answer
1  
Thank you, that's good, but aren't there included some junk words too? Like words of certain celebrities and popular topics of the year? –  xralf Jun 17 '12 at 16:58
    
Other thing is that a lot of words are two words like "stand up", "look up". Are these words correctly sorted according to frequencies? –  xralf Jun 17 '12 at 17:00
1  
@xralf Add those points to your original question. Add also any detail in order to scope this question enough, thanks. –  Alenanno Jun 17 '12 at 22:34
    
For reasons that need not be discussed, I searched for "trepidate" from 1800-1814 and found a number of references that are to magazines. –  Ross Millikan Jun 18 '12 at 4:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.