Tell me more ×
Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It's 100% free, no registration required.
up vote 6 down vote favorite

Most algorithms for splitting text into sentences which I've found rely on punctuation being correct. However, in many real world applications, there will be substantial numbers of punctuation errors (missing periods, extraneous periods, etc.) Are there sentence-splitting algorithms which deal with this?

share|improve this question
1  
For English speech, yes; intonation gives it away. For English spelling, no, not really. Not only doesn't English record intonational information reliably in its orthography, but what little does leak through into punctuation is quite inconsistently applied, and that's never going to change. Get used to culling ambiguous parse options of text through non-syntactic information. Like frames, for instance. – jlawler Feb 4 at 18:51
I was in fact imagining something like "if the likelihood of a sentence as given is too low, try splitting it in different reasonable places (or joining with next/previous sentence). If one of these splits increases likelihood substantially, repeat." – Alexey Romanov Feb 5 at 7:46
Or just trying to select and run a suitable machine learning algorithm, which would presumably assign high weights to punctuation and to capital letters, but could detect ends of sentence even if they are missing. The question is if this achieves enough accuracy... – Alexey Romanov Feb 5 at 7:57
1  
It'll give you enough semantics to translate roughly, but it's unlikely to give sentence boundaries. At least as they writer intended them. – jlawler Feb 5 at 23:20
1  
The more I think about it, though, just the change in number of options should give you most constituent boundaries, including sentences, regardless of punctuation. But you couldn't distinguish sentence boundaries from other constituent boundaries without more processing of the kinds of options. – jlawler Feb 10 at 23:58

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.