3

Many languages have inflectional or agglutinating morphology - they have words with multiple or many forms due to aspect, degree or comparison, gender, mood, number, tense, etc.

A number of languages do not indicate boundaries between words in their written form. I include in this category languages which indicate boundaries between all syllables. The languages of this type I know of are Burmese (Myanmar), Chinese, Dzongkha, Japanese, Khmer (Cambodian), Lao, Thai, Tibetan, and Vietnamese. Plus a number of minority languages of Southeast Asia such as Mon and Shan.

Please note that I am only asking about inflectional morphology and not about derivational morphology or compounds.

Segmentation for languages without word boundaries presents a challenge to natural language processing due to ambiguity. Lemmatization of inflected forms is also a challenge due to ambiguity.

The only language I can think of for which NLP must tackle both these problems at once in Japanese. But are there any others I've overlooked?

Improve this question
7
  • Maybe i do not fully understand "these problems" (coming from a mostly NLP background). why is modern Chinese not included?
    – Falcon
    May 4, 2017 at 12:05
  • @Falcon: Modern Chinese is neither inflecting nor agglutinating. It has no noun, verb, or adjective endings for tense, aspect, mood, person, number, comparative, superlative, etc. It has very few prefixes or suffixes. Most people would only think of -们. Any such things that the language uses are expressed by discrete words. This is harder than English but easier than Japanese.
    – hippietrail
    May 8, 2017 at 2:40
  • 1
    to address your question regarding the NLP challenge, character/subword-level modeling is gaining considerable popularity in NLP (as part of a state-of-the-art system arxiv.org/abs/1609.08144) and this can be naturally extended to east asian languages (such as Japanese ieeexplore.ieee.org/document/6289079).
    – Falcon
    May 8, 2017 at 22:31
  • 1
    I believe lay people think of 们 as a suffix but that linguists don't always say so. I've also heard it described as a noun which means something like "and others". Few who have actually looked into it believe in treating each character as a word. Most words are two-characters. The other particles you mention are never treated as prefixes and Chinese does not have tense. In any case the problem with Japanese is not with the kanji but the kana, which can be individual words, endings, parts of endings, etc. Those are subject to morphological and orthographic variation just like in English -s/-es.
    – hippietrail
    May 10, 2017 at 5:11
  • 1
    thank you for the clarification. I think i learned something :). hope you find the naive but data-driven approach of us NLP people interesting too.
    – Falcon
    May 11, 2017 at 15:15

1 Answer 1

Reset to default
6

Just for a beginning: ancient Greek and Latin did not indicate word boundaries. All the letters are evenly spaced. Sanskrit separates only at the end of a verse.

Improve this answer
7
  • Do you know which phase of Ancient Greek? I hadn't heard of this in Sanskrit before. Sanskrit has been written in many scripts though, is this something that happens only in Devanagari Sanskrit, another script, or is it common to Sanskrit independent of script?
    – hippietrail
    Sep 26, 2014 at 9:46
  • 3
    I do not think that Greek writing (whether in inscriptions or codices) regularly separates words until the Byzantine period. In, for example, the Vedas each entire verse is treated as a single unit for the purposes of saṁdhi. Each akṣara represents a syllable even across word boundaries (for example if a word ending with -t is followed by one beginning with a- you represent the two phonemes by a single akṣara <ta>). This happens regardless of which script you use.
    – fdb
    Sep 26, 2014 at 10:04
  • Interesting. I wonder what work has been done on NLP for unsegmented Latin and Ancient Greek. I expect they would be easier to work on than Japanese but of course of less use as they are dead languages that nobody investing in research will want to data-mine. I can find articles on Sanskrit NLP and I'm currently trying to find anything on segmenting verses into words, etc ...
    – hippietrail
    Sep 26, 2014 at 10:12
  • Of course, the whole concept of “word” is problematic, not least in a language like Greek that has lots of proclitics and postclitics forming an accentual unit with the word to which they are attached.
    – fdb
    Sep 26, 2014 at 10:21
  • 2
    @hippietrail: In old manuscripts all words run together, but modern Sanskrit printing has spaces where one word ends with a vowel and the next starts with a consonant, e.g. "पाण्डवाश्चैव किमकुर्वत", but not all words e.g rare to see "पाण्डवाश् चैव किम् अकुर्वत" (See Lanman's Notes on the externals of Indian books where he defends this further division.) (also common in transliteration: pāṇḍavāś caiva kim akurvata). And of course the result of sandhi is very rarely split: चैव caiva not written as च एव ca eva.
    – ShreevatsaR
    May 3, 2015 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.