0

Where can I find a 1TB or larger 1-gram corpus?

I am comparing certain structures for term-frequency calculation, but without a significantly large dataset I won't be able to show the non-theoretical speed gains.

5
  • how many words is 1TB?
    – user483
    Commented Mar 3, 2012 at 17:45
  • 1TB seems like a lot. Does it have to be entirely in one language?
    – prash
    Commented Mar 4, 2012 at 11:57
  • @jlovegren 1TB = 10^12 bytes = normally 5*10^11 or 10^12 characters depending on encoding
    – Louis Rhys
    Commented Mar 4, 2012 at 12:12
  • prash: Well all I am measuring is term-frequency, so I suppose language would be irrelevant.
    – A T
    Commented Mar 4, 2012 at 12:41
  • Wikipedia will give you around 15GB, and Gutenberg around 5-10GB
    – prash
    Commented Mar 4, 2012 at 12:53

2 Answers 2

2

Mark Davies' version of the Google Books corpus (has more functionality than the Ngram viewer) has 155 billion words, which is getting close to 1TB.

1

Are the corpora available from Google Books Ngram Viewer not the kind of thing you are looking for?

2
  • I was looking at that earlier, it's 1-gram are only in the 10s of GB range. I mean, I could work with larger 2-gram and 3-gram... and could also split them up into 1-gram...
    – A T
    Commented Mar 3, 2012 at 17:41
  • I was looking for their 1TB corpus which I knew about vaguely I while ago and don't know if this is what they call it now or if that's another thing \-: Commented Mar 3, 2012 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.