I need a free English language corpus with at least 15 million words. The corpus should contain one or more plain text files. There should be no tagging, just raw text. The corpus should be free. I would prefer if the corpus contained was for modern English, with a mixture of: tv, radio, film, news, fiction, technical etc., or better still, just plain everyday conversation, but this is not a requirement. I will be processing each sentence in the text with the python programming language. Can anyone direct me to such a resource.
1
-
1The question is vague because there are so many different corpora that fit your current description. You can start at hit.uib.no/corpora/sites.html. If you're going to be using Python, NLTK itself comes with a few sets: nltk.org/api/nltk.corpus.html– prash ♦Commented Aug 22, 2013 at 13:06
Add a comment
|
2 Answers
Corpora containing more than 15 million words are often not freely available due to copyright issues (such as the British National Corpus and the Corpus of Contemporary American English).
The open part of the American National Corpus (OANC) might fulfill your criteria. It contains almost 15 m. words, it's free, and contains conversations and other genres. The spoken part consists mainly of the telephone based Switchboard corpus. If you want more face to face conversations consider adding the Santa Barbara Corpus of Spoken American English.
The OANC comes in versions with different annotation schemes. If you take the version with the least amount of annotation you should be able to strip off all the annotations with a regex find and replace scheme in Python or Notepad++.
-
-
2Is MICASE available for download? I could only find the concordancer interface on the website.– robertCommented Aug 22, 2013 at 16:37
-
Here is a variety of language corpora with millions of sentences each: http://corpora.informatik.uni-leipzig.de/download.html
-
2Possibly paid as it requires written permission and a license. Commented Oct 31, 2015 at 7:58
-
1Under CC-BY-NC (Requires attribution, and only non-commercial).– danukerCommented Jul 8, 2019 at 19:17