4

I'm doing lemmatization as part of a classifier right now.

My question is, if it makes sense to first use a dictionary lookup, to find the lemmas (wordnet) and after that additionally, to apply a (porter-)stemmer to the word vector?

I have most lemmas after the first step of course, you never know, can the usage of a stemmer after the first step harm?

Improve this question

1 Answer 1

Reset to default
1

I suppose it depends on the task. If you don't often expect to see out-of-vocabulary words (i.e., words for which you don't have lemmata), there is no point in stemming—in fact, it might be harmful in that it might normalize two words of different meaning to the same stemmed string.

Yes, stemming can be harmful. It'd be best to try lemmatization vs. stemming in isolation and see how it affects classification performance in the end.

Improve this answer
3
  • Are there any statistics that you can reference to support this argument?
    – Sam
    Commented Nov 16, 2014 at 11:44
  • 2
    I ran the Porter Stemmer over the list of unique lemmata in WordNet. This produced 15 duplicates (out of ~5000 words). Notable duplicates: level, come, model, sent, signal, taxi.... But it looks like stemming afterwards probably won't hurt too much (not sure how it would help you, either, though).
    – Jon Gauthier
    Commented Nov 16, 2014 at 20:53
  • Oh that's still interesting, thanks for the test. I think I need to do some statistical analysis, though. I'm still not sure.
    – Sam
    Commented Nov 16, 2014 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.