3

In a statistical language model, should one tokenize compounded words through agglutination as words by themselves (unigrams) or should they be tokenized into n-grams?

Example:

  1. Staatspolizei (ISO 639-1: de) :
    a. unigram: staatspolizei
    b. ngram (bigram): (stāt, polizei).

  2. Purushottama (ISO 639-1: hi)
    a. unigram: purushottama
    b. ngram (bigram): (purusha, uttama).

Which of the modeling (a or b) is linguistically proper (will yield a better language model)?

4
  • What do you mean "should"? Why does it matter?
    – user6726
    May 16 '17 at 3:32
  • 1
    I plead to leave this question open, it is frequent enough in corpus linguistics (and different linguists have different opinions on the answer, but this does not make it an "opinion based" question, IMO). May 16 '17 at 9:53
  • purushottam is nearly exclusively used for the Hindu god Rama, while purush + uttam can be any man May 16 '17 at 10:56
  • @jknappen: Thank you for your support. I am unsure why this question has the risk of being closed as an "opinion based" question as this is a hot research topic. For reference, here is the same question with very interesting answers: quora.com/…
    – Subhobroto
    May 16 '17 at 22:01
2

(Disclaimer: I don't have much background in linguistics)

I think - if you are relating to agglutinantive and not polysynthetic languges - it would depend on what you are trying to build and the language too.

But,

Staatspolizei (Secret State Police) may occur around words like-

The (whereas polizei would require and adjective before e.g. The Secret-State-Police vs The secret Police),

found (... Staatspolizei found and staats polizei found can have ambiguous interpretations if the language allows it, XYZ found something and XYZ found by me)

etc.

changing it's use totally.

whereas purushottama (man + ultimate/best/ideal = ideal man) is as you can see a compound of Noun + Adj. in that order, in Hindi , Sanskrit and other languages where it might be used purush uttama asti (please ignore grammatical/morphological endings if wrong) and purushottama asti have different meanings; The man is ideal and The ideal man (is) exists.

also purush uttama karya karta hai and purushottam karya karta hai mean The man does the best work and The ideal man does work respectively (The Adj. might be used to modify verb occuring in the same position)

The above distinctions thus, show that treating them as unigrams would be better than treating them as n-grams as far as the semantic role is concerned.

Hope that helps! :)

2

For parsing: if the word is already in your 'dictionary' then treat it as a 'unigram'; but if it is new to you then treat it as an 'ngram'. If you are doing semantic statistics on a corpus then you may need to do both.

Not the answer you're looking for? Browse other questions tagged or ask your own question.