For example, in Chinese, a full stop is (U+3002) and a space is   (U+3000). I need to break paragraphs into sentences and I want to make it compatible with as many languages as possible. What other common languages use special punctuation or spaces?

  • Current techniques don't use such simple decision trees: consider how ambiguous the English "." is. The state-of-the-art sentence splitters use machine learning techniques. See class.coursera.org/nlp/lecture/5. – prash Jun 9 '14 at 22:55
  • In English, the only space character is U+0020 and the only sentence terminators are ., !, and ?. It doesn't matter how these characters are used, I just need possible spaces and possible sentence endings. – Leo Jiang Jun 10 '14 at 1:22
  • @Linksku I'm not sure that's true… – snailplane Jun 10 '14 at 5:01
  • 2
    This is a field of its own in NLP called "sentence boundary disambiguation" (SBD) or "sentence breaking" or "sentence segmentation". There are two relevant Wikipedia articles: 1, 2. As with many things in linguistics and NLP, it's not nearly as easy as you think. – hippietrail Jun 10 '14 at 9:57
  • 3
    Please try the sentence boundary iterators in the ICU library : userguide.icu-project.org/… . Of course, as other comments have pointed out, this can't give a perfect solution, but it might be good enough for your purposes. – b_jonas Jun 11 '14 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.