I would like to know how the relation of corpus size to observed ngrams is(especially unigrams). Since the word frequency follows a Zipf distribution, i would expect some kind of logarithmic relation as well, but has anyone done research on this particular question?
3
-
1Interesting question! I don't quite understand why you ask for unigrams. A unigram consists of a single word, so this is exactly what Zipf's law describes, isn't it?– robertCommented Sep 12, 2013 at 14:01
-
No, Zipfs law describes that for a given corpus, how the frequencies of words/ngrams are distributed. I want to know how the relation between corpus size and dictionary size behaves. Given a corpus of size n, how many distinct words are in that corpus? That distribution is what i am interested in. If that also follows Zipf's law - great! References would be nice, though.– kutschkemCommented Sep 12, 2013 at 14:22
-
Right, Zipf's law tells you for a given corpus size how you can derive the frequency of a word from the rank of its frequency and the other way around. And what you call dictionary size is just the number of word types in the corpus, if I understand you correctly. The number of word types is equal to the rank of the least frequent word type.– robertCommented Sep 12, 2013 at 15:13
Add a comment
|
1 Answer
Thanks to Robert for pointing me at this. Here is an approach to approximate the number of word types in a corpus. We assume that Zipf's law holds and the frequency of the second most common word type is 1/2 the frequency of the most common word type and so on.
Let N be the corpus size. Let maxfreq be the (relative) frequency of the most common word type. Then by Zipf's law, it should be
freq(w) = maxfreq / rank(w)
where rank tells us at what rank(w) the word type is with respect to its frequency, and freq(w) is the relative frequency of that word type.
The number of word types can be approximated by the maximal rank of words that have an absolute frequency higher than 1.
max rank(w)
s.t. N*freq(w) >= 1
that means that this maximum is N*maxfreq
So, if the most common word had a relative frequency of 20%, then the number of word types would also be roughly 20% of the corpus size. I think this will overestimate pretty much always.