In machine learning on text data (aka natural language processing), it's common to apply a stemming or lemmatization algorithm to the text.
However, sometimes you want to go a step further. For example, these algorithms (e.g. the Snowball stemming or WordNet lemmatization algorithms) will not conflate dentist
and dentistry
. From my limited linguistics/NLP knowledge, this is because they are morphologically different words, despite being very close etymologically and semantically.
Wikipedia tells me that the core word dentist
would be a "secondary root" shared between dentist
and dentistry
. Is there an algorithm for finding these secondary roots in English?
Some other examples:
flash
,flashy
->flash
wait
,waiter
->wait
Please note that I am not very knowledgeable in this are, so this could well be an XY problem, or a matter of not knowing the right search terms.
Update
this was discussed in a few places on Freenode. User dmiles on #nlp offered his own algorithm, which was to strip off common prefixes and suffixes (like -ly
), then match words against a large database of nouns.
For example, to process electricity
:
- Strip off
-ity
to obtainelectric
- Strip off
-ic
to obtainelectr
. - No more suffixes. Stop.
The result, electr
, nicely reflects the Latin electrum
from which it derives.
Likewise, to process wordy
:
- Strip off
-y
to obtainword
. - No more suffixes. Stop.
This seems serviceable in a lot of cases, and can be implemented somewhat efficiently in computer code with the right data structures. But it also seems pretty inelegant, and my linguistics knowledge falls far short of taking care of the many edge cases that are sure to arise.
For example, premonition
came to English with the pre-
already attached, so stripping it off might be considered an overreach of the algorithm. Whereas prefab
almost certainly should have its pre-
removed, to share a root with fabricate
(remove -ate
, then remove -ic
). You could probably handle this by checking against the database after each suffix-stripping pass, stopping at the longest match. But there might be problems with that method too.
Tl,dr: one guy on Freenode has an ad-hoc method that works for him (and apparently he's been refining it for decades), but I'd still hope to find something more principled.
foma
for a classic computational way to analyze morphology. But English refuses to commit to a single pattern and loves to draw words and morphemes from different languages. (Look at "octopi"! That's a root from one language with an inflection from another, neither of which is English, and neither of which is productive in English! Language is beautiful.)