-1

I am processing text samples of the following languages:

  1. Thai
  2. Khmer
  3. Japanese
  4. Chinese Simplified
  5. Chinese Traditional &
  6. Amharic

I need the text samples to be segmented as sentences using a regex.

As I am not familiar with any of the above languages, any help by providing a regex for sentence segmentation for the above language(s) or any leads to find them would be highly helpful.

Thanks

1 Answer 1

1

Here's one for Japanese to get you started:

[^!?。]*[!?。]

You didn't specify a language, but here's a Python implementation for reference:

import nltk

text = "奥行きも幅も数百メートル、最深85メートルの巨大なすり鉢状の大穴が、数千人の男たちでびっしりと埋め尽くされていた。あちこちに鉱石の入ったずだ袋が人の背丈ほどに積み上げられている。男たちと袋が密集し、岩肌や地面すら見えないほどだ。"

sentence_splitter = nltk.RegexpTokenizer(u'[^!?。]*[!?。]')
sents = sentence_splitter.tokenize(text)
for s in sents:
    print(s)
1
  • 1
    The same regex should work for both Simplified and Traditional Chinese as well, perhaps with the addition of . Commented Aug 16, 2022 at 7:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.