Most tutorials I've seen on language models illustrate MLE estimation by counting. For example,

P("mice"|"three blind") = count("three blind mice") / count("three blind")

But joint probability is commutative, and I'm pretty sure so is the context. P(c|a, b) = P(c|b, a)

I'm confused where this leaves MLE estimation though. Because I think the above means, P("mice"|"three", "blind") = P("mice"|"blind", "three"). In other words, P("three blind mice") = P("blind three mice") in this trigram model.

Is the above incorrect?

If it is correct, how then does MLE estimation work? Is the actual formula something like,

P("mice"|"three", "blind") = [count("three blind mice") + count("blind three mice)] / [count("three blind") + count("blind three")]

  • 1
    Some do, some don't. The general intuition is that bag-of-words is can do better when the dataset is smaller, otherwise it's needlessly lossy. Apr 4, 2018 at 17:55
  • 1
    Thanks for your comment! For what it's worth, I think I finally found an answer that dissolved my sense of confusion, stats.stackexchange.com/questions/102811/…
    – Kevin Binz
    Apr 4, 2018 at 17:59

1 Answer 1


While there are some context models that are unordered (Latent Semantic Analysis, some semantic word-vector approaches), that's not what you seem to be talking about.

Instead, I think the critical point is that you're describing the n-grams incorrectly. An n-gram model is usually ordered, so that the whole sequence "three blind mice" is a trigram, and "blind three mice" is a totally different trigram.

Your final formula seems to be trying to do both at once, which I'm not sure I've seen before. If you're really trying to answer the question 'What is P("mice") given that I know the last two words were "three" and "blind", but I don't know in what order?', then I think it would just be P("mice"|"three blind") + P("mice"|"blind three"). Obviously, this would get a lot more verbose at higher n-gram orders, but at that point I think you are indeed doing something closer to vector semantics.

You might review Speech and Language Processing by Daniel Jurafsky & James H. Martin, Chapter 4 'N-grams' (https://web.stanford.edu/~jurafsky/slp3/4.pdf), specifically Eq. 4.2; it's true that the focus on bigrams makes it slightly ambiguous that larger n-gram models are not ignoring word order.

  • 1
    Agree with your description of n-grams. Just want to point out that there are CBOW models as opposed to SG models. Apr 4, 2018 at 17:52
  • Thanks, it’s the edge of my experience so I didn’t want to add more. Apr 5, 2018 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.