7

I'm trying to make a dictionary on the web where people can automatically look up words in a text they're reading online. While I have the words and their definitions, the problem is that many times the word in the text won't be in it's root form - for example the word in the text may be המלכה (the queen) instead of מלכה (queen), where the latter is in the dictionary but the former is not. For nouns it doesn't seem too hard to drop the suffixes indicating plurals and/or the set of possible prefixes, but for verbs this seems like a tough job. Has any work been done on algorithms that can be applied to a word in Hebrew or Aramaic to obtain the root? Note that I'm asking for Hebrew and Aramaic because I believe that the languages are similar enough that what works for one should work for the other. Even if they do necessitate different languages, I will eventually want to do both, so info on either is appreciated.

  • 3
    The name of the field you are describing is lemmatization. A related field that's probably too simplistic for your needs, despite its name, is stemming. – hippietrail Aug 29 '14 at 15:14
  • 1
    @hippietrail thanks - that will certainly help when searching google and jstor. – PopularIsn'tRight Aug 29 '14 at 19:48
  • 1
    @Bachrach44 please do write back should you find anything useful for your purposes, so that others with a similar need can use it as well. Thanks! – Joe Pineda Sep 2 '14 at 2:00
  • @Bachrach44 you would most certainly find this book intriguing. – ylluminate Nov 23 '16 at 22:17
  • it's worth noting that it is not always possible to unambiguously get the consonantal root from a word (even after lemmatisation), especially in unvocalised text. A word beginning with a consonant that is a common preformative (like מ־ or ת־) may have those as radicals or as part of the pattern; likewise in many patterns, various different forms of weak roots may not be distinguished even in speech (or in vocalised text) and it requires careful comparison with other words in other patterns to work out. Lastly there's also the fact that many words don't even come from consonantal words at all – Tristan May 19 at 13:10
3

As mentioned above, you are looking for lemmatization tools for these languages. I admit to not knowing anything about Aramaic, but I do know that Hebrew is considered harder than English with regards to stemming/lemmatization (I'm afraid I don't have a credible source for this at the moment).

A potentially good place to start might be the MILA center for processing Hebrew (http://www.mila.cs.technion.ac.il/).

| improve this answer | |
  • It is quite likely harder to lemmatize Hebrew because of its mix between concatenative and non-concatenative morphology. – Typhon Oct 4 '14 at 16:34
  • If nothing else, I'd expect lemmatization to be harder in Hebrew because the spelling leaves out many more details than English spelling, such that patterns that are obvious in speech are not so obvious in writing, especially for an unknown word taken in isolation. – ruakh May 5 '18 at 1:01
1

There's a research program called "Bar Ilan Responsa Project." It's a database of tens of thousands of Hebrew and Aramaic works (constantly being updated and expanded), and it's fully searchable.

When conducting a search, there is the option to search by root, or the whole word, prefaces etc. So you could search for המלכה, מלכה, מ.ל.כ or other combinations.

Their search is limited to the database itself, so you can't search words online for it, but the technology that you're looking for definitely exists and I assume you could apply it to the web for your purpose as well.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.