I do understand that a lookup tagger is one of the simplest ones to implement as it is able to map each word a single POS. Why shouldn't we use it compared to bigram tagger?

  • There's no reason the lookup can't return compound POS codes (there's only about 30 in typical English). You can get pretty far using triplets of those to prune the possibilities (without using ngrams). It also helps to use a theory of syntax that doesn't require POS to change when there are syntactic elisions. – amI Dec 5 at 7:24
up vote 4 down vote accepted

Simply put, they're just not as good. Lookup taggers can't deal with the fact that words can have multiple parts of speech: look at "project" in English, which can be a verb or a noun. A lookup tagger can't distinguish between those two and will always choose whichever was most common in its training corpus.

N-gram taggers try to deal with this problem by adding context. But they quickly run into the Combinatorial Explosion: given all the possible ways words can be combined, it's easy to find N-grams in the wild that never appeared in the training corpus. So N-gram taggers tend to fall back to smaller and smaller values of N if they can't find the whole thing, eventually resorting to N=1: a lookup tagger.

  • Could you elaborate on what kinds of context n-gram taggers examine? They look at the adjacent words themselves, or do they look at features of those words only? For example, flower can be a noun or a verb, and so can pot. Verbs often follow nouns, so that might trip up something trying to look at the string "flower pot". What does an n-gram tagger do. – Wilson Dec 5 at 19:25

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.