Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on several occasions that trigram models outperform PCFGs.

What would be a good source to cite for trigram models being something of a standard?

What would be a good source to cite for the claim that trigram models outperform more complex models such as PCFGs?

I've gone ahead and started a bounty for this question: even just one good reference will help!

  • 1
    you might see if your library has this book: www-nlp.stanford.edu/fsnlp
    – user483
    Oct 6, 2012 at 17:35
  • @jlovegren Thanks, although I'm not sure that they actually compare trigram models to the others empirically in that book, or even argue that they outperform more complex language models. I'll read again more closely.
    – Julie
    Oct 10, 2012 at 21:52
  • I've never read the book, so you've probably gotten it right.
    – user483
    Oct 11, 2012 at 0:17

1 Answer 1


The reason why trigrams can be considered powerful compared to n-grams of higher order, lies in the problem of data sparsity; when n is higher, data becomes increasingly sparse. Because of this, using trigrams is a good compromise which often yields good results.

In a study by Chen and Goodman (1998), the effect of varying n-gram orders on the performance of a language model was compared. As expected, they found that 4-grams and 5-grams significantly outperform trigram models when using a very large data set (approximately 1e+06 sentences).

I'm not sure where you could find a source stating that trigrams outperform other models, but if you cite Chen and Goodman you can at least say that higher order n-grams are only viable with large data sets.

(Sorry I can't provide you with a link to this article, but here's the citation at least!)

Chen, Stanley F. and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. Harvard University, Center for Research in Computing Technology, TR-10-98:1–63, 1998.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.