Do language models ignore word order in the context?

Question

Most tutorials I've seen on language models illustrate MLE estimation by counting. For example,

P("mice"|"three blind") = count("three blind mice") / count("three blind")

But joint probability is commutative, and I'm pretty sure so is the context. P(c|a, b) = P(c|b, a)

I'm confused where this leaves MLE estimation though. Because I think the above means, P("mice"|"three", "blind") = P("mice"|"blind", "three"). In other words, P("three blind mice") = P("blind three mice") in this trigram model.

Is the above incorrect?

If it is correct, how then does MLE estimation work? Is the actual formula something like,

P("mice"|"three", "blind") = [count("three blind mice") + count("blind three mice)] / [count("three blind") + count("blind three")]

Answer 1

While there are some context models that are unordered (Latent Semantic Analysis, some semantic word-vector approaches), that's not what you seem to be talking about.

Instead, I think the critical point is that you're describing the n-grams incorrectly. An n-gram model is usually ordered, so that the whole sequence "three blind mice" is a trigram, and "blind three mice" is a totally different trigram.

Your final formula seems to be trying to do both at once, which I'm not sure I've seen before. If you're really trying to answer the question 'What is P("mice") given that I know the last two words were "three" and "blind", but I don't know in what order?', then I think it would just be P("mice"|"three blind") + P("mice"|"blind three"). Obviously, this would get a lot more verbose at higher n-gram orders, but at that point I think you are indeed doing something closer to vector semantics.

You might review Speech and Language Processing by Daniel Jurafsky & James H. Martin, Chapter 4 'N-grams' (https://web.stanford.edu/~jurafsky/slp3/4.pdf), specifically Eq. 4.2; it's true that the focus on bigrams makes it slightly ambiguous that larger n-gram models are not ignoring word order.