Said tags in Sentence detection

Question

I am trying to apply NLP to fiction but I don't know how split dialog into sentences. Specifically how to split said tags. ex.

"Awake? How long have been asleep?" asked teenagers at the same time.

do I split it like this:

"Awake?
How long have been asleep?"
asked teenagers at the same time.

or like this:

"Awake?
How long have been asleep?" asked teenagers at the same time.

or was it already one sentence

Answer 1

In my opinion it should be one sentence, but it could also depend on your use case. For example, if you are extracting sentences for the purpose of translation, I think the example you provided should be one sentence.

I wrote a sentence segmenter and tried to find all of the "edge cases" like the example you provided. I then tested all of the popular tools against these edge cases. If you would like to read more about it please check the README here.

You can also give your example a try in my tool here (there is a browser-based version part way down the page that lets you enter some text) and you'll see it segments it as one sentence.

Answer 2

From the question, it is not clear which language you are using or how you want the sentence to be split.

You're looking for a sentence detector, though. If you're using Python, you can use NLTK's sent_detector module. By default, it will break up your sentence as:

>>> import nltk
>>> sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
>>> txt = '"Awake? How long have been asleep?" asked teenagers at the same time.'
>>> sent_detector.tokenize(txt)
['"Awake?', 'How long have been asleep?', '" asked teenagers at the same time.']

If the placement of the quotation marks is important, you can do:

>>> sent_detector.tokenize(txt.strip(), realign_boundaries=True)
['"Awake?', 'How long have been asleep?"', 'asked teenagers at the same time.']

You can also train a model, using punkt. In this case, you would feed it examples split up the way you like, and it would learn the pattern.