In Estonian, "in London" is translated as "Londonis". Now if we want to apply natural language understanding to this, then should I tokenize "London" and "is" separately? Or should I lemmatize "Londonis" to "London"? Ofcourse both approaches might be possible, but I'm asking for the rational and recommended approach. The issue with words with suffixes is that the sequence tagging statistical model for named entity recognition is not detecting words with suffixes. And I don't speak Estonian.
-
Is "in Paris" translated "Parisis" ? If yes, so you should consider is as morpheme and tokenize it separately. – amegnunsen Feb 25 '19 at 10:54
-
Yes, "in Paris" is translated as "Pariisis", because the name is "Pariis" in estonian – Hasan Shaukat Feb 25 '19 at 11:03
-
1It depends very much on your approach to Natural Language Understanding. When you use a rule-based approach, you should definitely keep all morphemes. Using a character based neural network approach, you don't even think about morphemes. – jk - Reinstate Monica Feb 25 '19 at 11:07
-
I intend to use word based understanding. – Hasan Shaukat Feb 25 '19 at 11:14
-
And what do you do with the words in order to get understanding? Do you train a statistical model, do you apply rukes or a neural network? Please add that information to your question (you can always edit your own questions, even with only 1 point of reputation). – jk - Reinstate Monica Feb 25 '19 at 12:39
Add a comment
|
