Where can I find a 1TB or larger 1-gram corpus?
I am comparing certain structures for term-frequency calculation, but without a significantly large dataset I won't be able to show the non-theoretical speed gains.
-
how many words is 1TB?– user483Mar 3, 2012 at 17:45
-
1TB seems like a lot. Does it have to be entirely in one language?– prash ♦Mar 4, 2012 at 11:57
-
@jlovegren 1TB = 10^12 bytes = normally 5*10^11 or 10^12 characters depending on encoding– Louis RhysMar 4, 2012 at 12:12
-
prash: Well all I am measuring is term-frequency, so I suppose language would be irrelevant.– A TMar 4, 2012 at 12:41
-
Wikipedia will give you around 15GB, and Gutenberg around 5-10GB– prash ♦Mar 4, 2012 at 12:53
Add a comment
|
2 Answers
Mark Davies' version of the Google Books corpus (has more functionality than the Ngram viewer) has 155 billion words, which is getting close to 1TB.
Are the corpora available from Google Books Ngram Viewer not the kind of thing you are looking for?
answered Mar 3, 2012 at 17:05
-
I was looking at that earlier, it's 1-gram are only in the 10s of GB range. I mean, I could work with larger 2-gram and 3-gram... and could also split them up into 1-gram...– A TMar 3, 2012 at 17:41
-
I was looking for their 1TB corpus which I knew about vaguely I while ago and don't know if this is what they call it now or if that's another thing \-: Mar 3, 2012 at 17:51