I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities associated with 2-grams than derived 1-grams. For example, the log probabilities of "sick" and "feel sick":
sick : -4.48 feel sick : -2.6995
Can anyone explain why this occurs? I would have thought that the probability of a single word in a text would be higher than a pair of words in the same text?
For example in the following text, not including punctuation:
I feel happy, so very happy. You make me very happy.
11 1-grams 9 2-grams
"happy" 3/11 = 0.27 "very happy" 2/9 = 0.22
I find it hard to think of a situation where a 2-gram would be more probable than a 1-gram contained within the 2-gram.