Take the 2-minute tour ×
Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It's 100% free, no registration required.

Many people in computational linguistics seem to mention the unexpected power of trigram (or 2nd order Markov) models for language modeling. For instance, it has been stated (verbally) to me on several occasions that trigram models outperform PCFGs.

What would be a good source to cite for trigram models being something of a standard?

What would be a good source to cite for the claim that trigram models outperform more complex models such as PCFGs?

I've gone ahead and started a bounty for this question: even just one good reference will help!

share|improve this question
you might see if your library has this book: www-nlp.stanford.edu/fsnlp –  jlovegren Oct 6 '12 at 17:35
@jlovegren Thanks, although I'm not sure that they actually compare trigram models to the others empirically in that book, or even argue that they outperform more complex language models. I'll read again more closely. –  Julie Oct 10 '12 at 21:52
I've never read the book, so you've probably gotten it right. –  jlovegren Oct 11 '12 at 0:17
add comment

1 Answer

The reason why trigrams can be considered powerful compared to n-grams of higher order, lies in the problem of data sparsity; when n is higher, data becomes increasingly sparse. Because of this, using trigrams is a good compromise which often yields good results.

In a study by Chen and Goodman (1998), the effect of varying n-gram orders on the performance of a language model was compared. As expected, they found that 4-grams and 5-grams significantly outperform trigram models when using a very large data set (approximately 1e+06 sentences).

I'm not sure where you could find a source stating that trigrams outperform other models, but if you cite Chen and Goodman you can at least say that higher order n-grams are only viable with large data sets.

(Sorry I can't provide you with a link to this article, but here's the citation at least!)

Chen, Stanley F. and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. Harvard University, Center for Research in Computing Technology, TR-10-98:1–63, 1998.

share|improve this answer
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.