In Estonian, "in London" is translated as "Londonis". Now if we want to apply natural language understanding to this, then should I tokenize "London" and "is" separately? Or should I lemmatize "Londonis" to "London"? Ofcourse both approaches might be possible, but I'm asking for the rational and recommended approach. The issue with words with suffixes is that the sequence tagging statistical model for named entity recognition is not detecting words with suffixes. And I don't speak Estonian.

  • Is "in Paris" translated "Parisis" ? If yes, so you should consider is as morpheme and tokenize it separately.
    – amegnunsen
    Feb 25, 2019 at 10:54
  • Yes, "in Paris" is translated as "Pariisis", because the name is "Pariis" in estonian Feb 25, 2019 at 11:03
  • 1
    It depends very much on your approach to Natural Language Understanding. When you use a rule-based approach, you should definitely keep all morphemes. Using a character based neural network approach, you don't even think about morphemes. Feb 25, 2019 at 11:07
  • I intend to use word based understanding. Feb 25, 2019 at 11:14
  • And what do you do with the words in order to get understanding? Do you train a statistical model, do you apply rukes or a neural network? Please add that information to your question (you can always edit your own questions, even with only 1 point of reputation). Feb 25, 2019 at 12:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.