1

In machine learning on text data (aka natural language processing), it's common to apply a stemming or lemmatization algorithm to the text.

However, sometimes you want to go a step further. For example, these algorithms (e.g. the Snowball stemming or WordNet lemmatization algorithms) will not conflate dentist and dentistry. From my limited linguistics/NLP knowledge, this is because they are morphologically different words, despite being very close etymologically and semantically.

Wikipedia tells me that the core word dentist would be a "secondary root" shared between dentist and dentistry. Is there an algorithm for finding these secondary roots in English?

Some other examples:

  • flash, flashy -> flash
  • wait, waiter -> wait

Please note that I am not very knowledgeable in this are, so this could well be an XY problem, or a matter of not knowing the right search terms.

Update

this was discussed in a few places on Freenode. User dmiles on #nlp offered his own algorithm, which was to strip off common prefixes and suffixes (like -ly), then match words against a large database of nouns.

For example, to process electricity:

  • Strip off -ity to obtain electric
  • Strip off -ic to obtain electr.
  • No more suffixes. Stop.

The result, electr, nicely reflects the Latin electrum from which it derives.

Likewise, to process wordy:

  • Strip off -y to obtain word.
  • No more suffixes. Stop.

This seems serviceable in a lot of cases, and can be implemented somewhat efficiently in computer code with the right data structures. But it also seems pretty inelegant, and my linguistics knowledge falls far short of taking care of the many edge cases that are sure to arise.

For example, premonition came to English with the pre- already attached, so stripping it off might be considered an overreach of the algorithm. Whereas prefab almost certainly should have its pre- removed, to share a root with fabricate (remove -ate, then remove -ic). You could probably handle this by checking against the database after each suffix-stripping pass, stopping at the longest match. But there might be problems with that method too.

Tl,dr: one guy on Freenode has an ad-hoc method that works for him (and apparently he's been refining it for decades), but I'd still hope to find something more principled.

  • 1
    Unfortunately a whole lot of English morphology only makes sense in Latin: consider "vortex" and "convert", which come from the same Latin root but look sufficiently different in English that most algorithms would fail to connect them. Should they be connected? They're just as close as "pendulous" and "pendulum"…and "depend", and "pensive", and "impending". – Draconis Oct 30 '18 at 3:43
  • 1
    Basically, English morphology is a beautiful mess. Other languages would make this easy; look into foma for a classic computational way to analyze morphology. But English refuses to commit to a single pattern and loves to draw words and morphemes from different languages. (Look at "octopi"! That's a root from one language with an inflection from another, neither of which is English, and neither of which is productive in English! Language is beautiful.) – Draconis Oct 30 '18 at 3:46
  • Thanks for that. I'm actually starting to think that this iterative pre/suffix stripping isn't so bad. If it helps, the original intent of this was to consolidate words for input to machine learning algorithms; the idea is that English words constructed by tacking stuff onto a common root should be semantically related. "Vortex" and "convert" don't fit that pattern, and I also wouldn't say that they are topically related. As for "octopus", I would only want to ensure that "octopus" and "octopuslike" are reduced to a common root. I will also look into foma. – shadowtalker Oct 30 '18 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.