1

I want to train a german embedding and need to split text into sentences. That is not easy since "z. B." and "Dr." are not endings of a sentence. Does anybody know a tool to do that for german texts? Thanks.

Improve this question

3 Answers 3

Reset to default
2

Just searching for tokenizer german I came up with Stefanie Dipper's tokenizer for German. It comes with a somewhat non-commercial license (read it before download).

Improve this answer
2
  • It works but this question is really sentence boundary detection not tokenisation.
    – Adam Bittlingmayer
    Commented Oct 19, 2018 at 18:42
  • It is usual that a tokenizer does sentence boundary detection as well and that there is no separate tool for this task.
    – Sir Cornflakes
    Commented Oct 19, 2018 at 19:05
2

spaCy does sentence boundary detection (sentence segmentation, sentence breaking). It also supports German.

See https://spacy.io/api/doc#sents for an example:

enter image description here

Improve this answer
1

I found this here to be the best solution for my problem: https://github.com/tsproisl/SoMaJo

I compared it to spaCy and NLTK but SoMaJo was better quality in sentence splitting the german texts.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.