1

I want to train a german embedding and need to split text into sentences. That is not easy since "z. B." and "Dr." are not endings of a sentence. Does anybody know a tool to do that for german texts? Thanks.

2

Just searching for tokenizer german I came up with Stefanie Dipper's tokenizer for German. It comes with a somewhat non-commercial license (read it before download).

2
  • It works but this question is really sentence boundary detection not tokenisation. – Adam Bittlingmayer Oct 19 '18 at 18:42
  • It is usual that a tokenizer does sentence boundary detection as well and that there is no separate tool for this task. – jk - Reinstate Monica Oct 19 '18 at 19:05
2

spaCy does sentence boundary detection (sentence segmentation, sentence breaking). It also supports German.

See https://spacy.io/api/doc#sents for an example:

enter image description here

1

I found this here to be the best solution for my problem: https://github.com/tsproisl/SoMaJo

I compared it to spaCy and NLTK but SoMaJo was better quality in sentence splitting the german texts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.