# Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities associated with 2-grams than derived 1-grams. For example, the log probabilities of "sick" and "feel sick":

``````sick : -4.48
feel sick : -2.6995
``````

Can anyone explain why this occurs? I would have thought that the probability of a single word in a text would be higher than a pair of words in the same text?

For example in the following text, not including punctuation:

``````I feel happy, so very happy.  You make me very happy.
``````

There are:

``````11 1-grams
9  2-grams
``````

Giving probabilities:

``````"happy" 3/11 = 0.27
"very happy" 2/9 = 0.22
``````

I find it hard to think of a situation where a 2-gram would be more probable than a 1-gram contained within the 2-gram.

• There is a proposal (as at July 2014) for a Natural Language Processing stack exchange which would suit this type of question, please follow it! area51.stackexchange.com/proposals/74384/… Jul 23, 2014 at 2:44
• the NLP stack-exchange didn't get enough traction. Feb 27, 2015 at 14:23

This could occur when the single word is much more commonly used in a set phrase or idiom than on its own. For example, it is much more common to see the word 'eke' with an 'out' than it standing alone.

• I have added an example to the question to show why I don't think this is the case. Jul 23, 2014 at 2:42

I'm somewhat new to NLP, so if someone else has a more complete answer, please go ahead and clarify.

What I believe is going on here is that the 2-gram (and 3-gram etc.) probabilities are actually conditional probabilities, while the 1-gram probabilities are unconditional probabilities.

So in your example, the probability of `sick` is `10^-4.48` = 0.00003311, or a 1 in 30200 chance of occurring. On the other hand, the probability of `sick` given that `feel` precedes it is `10^-2.6995` = 0.00199, or a 1 in 500 chance.

In other words, `P(sick)` = `10^-4.48`, and `P(sick|feel)` = `10^-2.6995`.

• Thanks for the answer, do you have a link to anything that supports your argument? Jan 12, 2015 at 0:32