I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities associated with 2-grams than derived 1-grams. For example, the log probabilities of "sick" and "feel sick":

sick : -4.48
feel sick : -2.6995

Can anyone explain why this occurs? I would have thought that the probability of a single word in a text would be higher than a pair of words in the same text?

For example in the following text, not including punctuation:

I feel happy, so very happy.  You make me very happy.

There are:

11 1-grams
9  2-grams

Giving probabilities:

"happy" 3/11 = 0.27
"very happy" 2/9 = 0.22

I find it hard to think of a situation where a 2-gram would be more probable than a 1-gram contained within the 2-gram.


2 Answers 2


This could occur when the single word is much more commonly used in a set phrase or idiom than on its own. For example, it is much more common to see the word 'eke' with an 'out' than it standing alone.

  • 1
    I have added an example to the question to show why I don't think this is the case.
    – kristianp
    Jul 23, 2014 at 2:42

I'm somewhat new to NLP, so if someone else has a more complete answer, please go ahead and clarify.

What I believe is going on here is that the 2-gram (and 3-gram etc.) probabilities are actually conditional probabilities, while the 1-gram probabilities are unconditional probabilities.

So in your example, the probability of sick is 10^-4.48 = 0.00003311, or a 1 in 30200 chance of occurring. On the other hand, the probability of sick given that feel precedes it is 10^-2.6995 = 0.00199, or a 1 in 500 chance.

In other words, P(sick) = 10^-4.48, and P(sick|feel) = 10^-2.6995.

  • Thanks for the answer, do you have a link to anything that supports your argument?
    – kristianp
    Jan 12, 2015 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.