In Estonian, "in London" is translated as "Londonis". Now if we want to apply natural language understanding to this, then should I tokenize "London" and "is" separately? Or should I lemmatize "Londonis" to "London"? Ofcourse both approaches might be possible, but I'm asking for the rational and recommended approach. The issue with words with suffixes is that the sequence tagging statistical model for named entity recognition is not detecting words with suffixes. And I don't speak Estonian.
-
Is "in Paris" translated "Parisis" ? If yes, so you should consider is as morpheme and tokenize it separately.– amegnunsenFeb 25, 2019 at 10:54
-
Yes, "in Paris" is translated as "Pariisis", because the name is "Pariis" in estonian– Hasan ShaukatFeb 25, 2019 at 11:03
-
1It depends very much on your approach to Natural Language Understanding. When you use a rule-based approach, you should definitely keep all morphemes. Using a character based neural network approach, you don't even think about morphemes.– jk - Reinstate MonicaFeb 25, 2019 at 11:07
-
I intend to use word based understanding.– Hasan ShaukatFeb 25, 2019 at 11:14
-
And what do you do with the words in order to get understanding? Do you train a statistical model, do you apply rukes or a neural network? Please add that information to your question (you can always edit your own questions, even with only 1 point of reputation).– jk - Reinstate MonicaFeb 25, 2019 at 12:39
Add a comment
|