# Do language models ignore word order in the context?

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example,

P("mice"|"three blind") = count("three blind mice") / count("three blind")

But joint probability is commutative, and I'm pretty sure so is the context. P(c|a, b) = P(c|b, a)

I'm confused where this leaves MLE estimation though. Because I think the above means, P("mice"|"three", "blind") = P("mice"|"blind", "three"). In other words, P("three blind mice") = P("blind three mice") in this trigram model.

Is the above incorrect?

If it is correct, how then does MLE estimation work? Is the actual formula something like,

P("mice"|"three", "blind") = [count("three blind mice") + count("blind three mice)] / [count("three blind") + count("blind three")]

• Some do, some don't. The general intuition is that bag-of-words is can do better when the dataset is smaller, otherwise it's needlessly lossy. Apr 4 '18 at 17:55
• Thanks for your comment! For what it's worth, I think I finally found an answer that dissolved my sense of confusion, stats.stackexchange.com/questions/102811/… Apr 4 '18 at 17:59