4

I'm doing lemmatization as part of a classifier right now.

My question is, if it makes sense to first use a dictionary lookup, to find the lemmas (wordnet) and after that additionally, to apply a (porter-)stemmer to the word vector?

I have most lemmas after the first step of course, you never know, can the usage of a stemmer after the first step harm?

1

I suppose it depends on the task. If you don't often expect to see out-of-vocabulary words (i.e., words for which you don't have lemmata), there is no point in stemming—in fact, it might be harmful in that it might normalize two words of different meaning to the same stemmed string.

Yes, stemming can be harmful. It'd be best to try lemmatization vs. stemming in isolation and see how it affects classification performance in the end.

| improve this answer | |
  • Are there any statistics that you can reference to support this argument? – Sam Nov 16 '14 at 11:44
  • 2
    I ran the Porter Stemmer over the list of unique lemmata in WordNet. This produced 15 duplicates (out of ~5000 words). Notable duplicates: level, come, model, sent, signal, taxi.... But it looks like stemming afterwards probably won't hurt too much (not sure how it would help you, either, though). – Jon Gauthier Nov 16 '14 at 20:53
  • Oh that's still interesting, thanks for the test. I think I need to do some statistical analysis, though. I'm still not sure. – Sam Nov 16 '14 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.