28

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I would prefer if the corpus contained was for modern English, with a mixture of: tv, radio, film, news, fiction, technical etc., or better still, just plain everyday conversation, but this is not a requirement. I will be processing each sentence in the text with the python programming language. Can anyone direct me to such a resource.

1
22

Corpora containing more than 15 million words are often not freely available due to copyright issues (such as the British National Corpus and the Corpus of Contemporary American English).

The open part of the American National Corpus (OANC) might fulfill your criteria. It contains almost 15 m. words, it's free, and contains conversations and other genres. The spoken part consists mainly of the telephone based Switchboard corpus. If you want more face to face conversations consider adding the Santa Barbara Corpus of Spoken American English.

The OANC comes in versions with different annotation schemes. If you take the version with the least amount of annotation you should be able to strip off all the annotations with a regex find and replace scheme in Python or Notepad++.

3
  • There's also MICASE.
    – jlawler
    Aug 22 '13 at 16:28
  • 2
    Is MICASE available for download? I could only find the concordancer interface on the website.
    – robert
    Aug 22 '13 at 16:37
  • Contact them and ask.
    – jlawler
    Aug 22 '13 at 17:22
12

Here is a variety of language corpora with millions of sentences each: http://corpora.informatik.uni-leipzig.de/download.html

2
  • 2
    Possibly paid as it requires written permission and a license.
    – giorgio79
    Oct 31 '15 at 7:58
  • 1
    Under CC-BY-NC (Requires attribution, and only non-commercial).
    – danuker
    Jul 8 '19 at 19:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.