Where can I find a 1TB or larger 1-gram corpus?

I am comparing certain structures for term-frequency calculation, but without a significantly large dataset I won't be able to show the non-theoretical speed gains.

  • how many words is 1TB?
    – user483
    Mar 3 '12 at 17:45
  • 1TB seems like a lot. Does it have to be entirely in one language?
    – prash
    Mar 4 '12 at 11:57
  • @jlovegren 1TB = 10^12 bytes = normally 5*10^11 or 10^12 characters depending on encoding
    – Louis Rhys
    Mar 4 '12 at 12:12
  • prash: Well all I am measuring is term-frequency, so I suppose language would be irrelevant.
    – A T
    Mar 4 '12 at 12:41
  • Wikipedia will give you around 15GB, and Gutenberg around 5-10GB
    – prash
    Mar 4 '12 at 12:53

Mark Davies' version of the Google Books corpus (has more functionality than the Ngram viewer) has 155 billion words, which is getting close to 1TB.


Are the corpora available from Google Books Ngram Viewer not the kind of thing you are looking for?

  • I was looking at that earlier, it's 1-gram are only in the 10s of GB range. I mean, I could work with larger 2-gram and 3-gram... and could also split them up into 1-gram...
    – A T
    Mar 3 '12 at 17:41
  • I was looking for their 1TB corpus which I knew about vaguely I while ago and don't know if this is what they call it now or if that's another thing \-: Mar 3 '12 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.