Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

Question

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities associated with 2-grams than derived 1-grams. For example, the log probabilities of "sick" and "feel sick":

sick : -4.48
feel sick : -2.6995

Can anyone explain why this occurs? I would have thought that the probability of a single word in a text would be higher than a pair of words in the same text?

For example in the following text, not including punctuation:

I feel happy, so very happy.  You make me very happy.

There are:

11 1-grams
9  2-grams

Giving probabilities:

"happy" 3/11 = 0.27
"very happy" 2/9 = 0.22

I find it hard to think of a situation where a 2-gram would be more probable than a 1-gram contained within the 2-gram.

Answer 1

This could occur when the single word is much more commonly used in a set phrase or idiom than on its own. For example, it is much more common to see the word 'eke' with an 'out' than it standing alone.

Answer 2

I'm somewhat new to NLP, so if someone else has a more complete answer, please go ahead and clarify.

What I believe is going on here is that the 2-gram (and 3-gram etc.) probabilities are actually conditional probabilities, while the 1-gram probabilities are unconditional probabilities.

So in your example, the probability of sick is 10^-4.48 = 0.00003311, or a 1 in 30200 chance of occurring. On the other hand, the probability of sick given that feel precedes it is 10^-2.6995 = 0.00199, or a 1 in 500 chance.

In other words, P(sick) = 10^-4.48, and P(sick|feel) = 10^-2.6995.