1

In Estonian, "in London" is translated as "Londonis". Now if we want to apply natural language understanding to this, then should I tokenize "London" and "is" separately? Or should I lemmatize "Londonis" to "London"? Ofcourse both approaches might be possible, but I'm asking for the rational and recommended approach. The issue with words with suffixes is that the sequence tagging statistical model for named entity recognition is not detecting words with suffixes. And I don't speak Estonian.

Improve this question
5
  • Is "in Paris" translated "Parisis" ? If yes, so you should consider is as morpheme and tokenize it separately.
    – amegnunsen
    Feb 25 '19 at 10:54
  • Yes, "in Paris" is translated as "Pariisis", because the name is "Pariis" in estonian
    – Hasan Shaukat
    Feb 25 '19 at 11:03
  • 1
    It depends very much on your approach to Natural Language Understanding. When you use a rule-based approach, you should definitely keep all morphemes. Using a character based neural network approach, you don't even think about morphemes.
    – jk - Reinstate Monica
    Feb 25 '19 at 11:07
  • I intend to use word based understanding.
    – Hasan Shaukat
    Feb 25 '19 at 11:14
  • And what do you do with the words in order to get understanding? Do you train a statistical model, do you apply rukes or a neural network? Please add that information to your question (you can always edit your own questions, even with only 1 point of reputation).
    – jk - Reinstate Monica
    Feb 25 '19 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.