I'm trying to make a dictionary on the web where people can automatically look up words in a text they're reading online. While I have the words and their definitions, the problem is that many times the word in the text won't be in it's root form - for example the word in the text may be המלכה (the queen) instead of מלכה (queen), where the latter is in the dictionary but the former is not. For nouns it doesn't seem too hard to drop the suffixes indicating plurals and/or the set of possible prefixes, but for verbs this seems like a tough job. Has any work been done on algorithms that can be applied to a word in Hebrew or Aramaic to obtain the root? Note that I'm asking for Hebrew and Aramaic because I believe that the languages are similar enough that what works for one should work for the other. Even if they do necessitate different languages, I will eventually want to do both, so info on either is appreciated.
-
3The name of the field you are describing is lemmatization. A related field that's probably too simplistic for your needs, despite its name, is stemming.– hippietrailCommented Aug 29, 2014 at 15:14
-
1@hippietrail thanks - that will certainly help when searching google and jstor.– PopularIsn'tRightCommented Aug 29, 2014 at 19:48
-
1@Bachrach44 please do write back should you find anything useful for your purposes, so that others with a similar need can use it as well. Thanks!– Joe PinedaCommented Sep 2, 2014 at 2:00
-
@Bachrach44 you would most certainly find this book intriguing.– ylluminateCommented Nov 23, 2016 at 22:17
-
it's worth noting that it is not always possible to unambiguously get the consonantal root from a word (even after lemmatisation), especially in unvocalised text. A word beginning with a consonant that is a common preformative (like מ־ or ת־) may have those as radicals or as part of the pattern; likewise in many patterns, various different forms of weak roots may not be distinguished even in speech (or in vocalised text) and it requires careful comparison with other words in other patterns to work out. Lastly there's also the fact that many words don't even come from consonantal words at all– TristanCommented May 19, 2020 at 13:10
2 Answers
As mentioned above, you are looking for lemmatization tools for these languages. I admit to not knowing anything about Aramaic, but I do know that Hebrew is considered harder than English with regards to stemming/lemmatization (I'm afraid I don't have a credible source for this at the moment).
A potentially good place to start might be the MILA center for processing Hebrew (http://www.mila.cs.technion.ac.il/).
-
It is quite likely harder to lemmatize Hebrew because of its mix between concatenative and non-concatenative morphology.– TyphonCommented Oct 4, 2014 at 16:34
-
1If nothing else, I'd expect lemmatization to be harder in Hebrew because the spelling leaves out many more details than English spelling, such that patterns that are obvious in speech are not so obvious in writing, especially for an unknown word taken in isolation.– ruakhCommented May 5, 2018 at 1:01
There's a research program called "Bar Ilan Responsa Project." It's a database of tens of thousands of Hebrew and Aramaic works (constantly being updated and expanded), and it's fully searchable.
When conducting a search, there is the option to search by root, or the whole word, prefaces etc. So you could search for המלכה, מלכה, מ.ל.כ or other combinations.
Their search is limited to the database itself, so you can't search words online for it, but the technology that you're looking for definitely exists and I assume you could apply it to the web for your purpose as well.
