24

I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I would prefer if the corpus contained was for modern English, with a mixture of: tv, radio, film, news, fiction, technical etc., or better still, just plain everyday conversation, but this is not a requirement. I will be processing each sentence in the text with the python programming language. Can anyone direct me to such a resource.

19

Corpora containing more than 15 million words are often not freely available due to copyright issues (such as the British National Corpus and the Corpus of Contemporary American English).

The open part of the American National Corpus (OANC) might fulfill your criteria. It contains almost 15 m. words, it's free, and contains conversations and other genres. The spoken part consists mainly of the telephone based Switchboard corpus. If you want more face to face conversations consider adding the Santa Barbara Corpus of Spoken American English.

The OANC comes in versions with different annotation schemes. If you take the version with the least amount of annotation you should be able to strip off all the annotations with a regex find and replace scheme in Python or Notepad++.

  • There's also MICASE. – jlawler Aug 22 '13 at 16:28
  • 2
    Is MICASE available for download? I could only find the concordancer interface on the website. – robert Aug 22 '13 at 16:37
  • Contact them and ask. – jlawler Aug 22 '13 at 17:22
12

Here is a variety of language corpora with millions of sentences each: http://corpora.informatik.uni-leipzig.de/download.html

  • 1
    Possibly paid as it requires written permission and a license. – giorgio79 Oct 31 '15 at 7:58
  • Under CC-BY-NC (Requires attribution, and only non-commercial). – danuker Jul 8 '19 at 19:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.