6

There is a lot of easily located material online which describes stemmers or provides implementations. There's a bit of discussion of metrics for quantifying how effective they are, and I've even found an evaluation program. What I can't find anywhere is some manually tagged data for use in such evaluation.

Are there standard test data sets for English stemmers? (Test data sets for lemmatisers would obviously also serve). Are they freely available online?

2
  • Why do you want tagged data? You can use normal corpora. – Andrew Ravus Mar 7 '16 at 10:18
  • @Adel, I think you must understand something different by "tagged data" to what I'm trying to communicate. What I wanted was essentially a list of pairs of the form (word, lemma) which would allow me to test whether the stemmer gave the same output for every word which shares a lemma. – Peter Taylor Mar 7 '16 at 10:51
3

There is a list of vocabulary and the output from testing the snowball algorithm available at the Snowball Github page, though it is not precisely human tested but it's good for testing you stemmer.

It supports the following languages:

arabic  
danish
dutch
english
finnish
french  
german  
hungarian   
italian     
kraaij_pohlmann     
lovins
norwegian
portuguese 
romanian 
russian 
spanish 
swedish
tamil
turkish

You can find this dataset at: https://github.com/snowballstem/snowball-data

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.