Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It only takes a minute to sign up.
Sign up to join this community
There is a lot of easily located material online which describes stemmers or provides implementations. There's a bit of discussion of metrics for quantifying how effective they are, and I've even found an evaluation program. What I can't find anywhere is some manually tagged data for use in such evaluation.
Are there standard test data sets for English stemmers? (Test data sets for lemmatisers would obviously also serve). Are they freely available online?
There is a list of vocabulary and the output from testing the snowball algorithm available at the Snowball Github page, though it is not precisely human tested but it's good for testing you stemmer.
It supports the following languages:
arabic
danish
dutch
english
finnish
french
german
hungarian
italian
kraaij_pohlmann
lovins
norwegian
portuguese
romanian
russian
spanish
swedish
tamil
turkish
You can find this dataset at: https://github.com/snowballstem/snowball-data
