I'm writing a tokenizer/parser thing to parse & diagram a bunch of sentences gathered from ordinary language use. Most of it parses just fine by following rules, but from time to time there are errors in the text.

What strategies do linguistis use to deal with mistakes in the corpus? Do we kick out the mistakes or fix them, which feels like tampering with the data, or do we extend the rules to account for "recoverable mistakes" (the must be recoverable or else even I wouldn't be able to figure out what it means). Or is there some other strategy?

  • 1
    If you have a very specific purpose for your corpus (such as topic recognition, named entity recognition) etc. it may be useful to correct errors. But if you want to address questions regarding language structure or a specific language or variety, "correcting" errors would indeed be tampering with the data. – robert Oct 27 '14 at 13:39

Science must be reproducible, and closely connected to experimental data. This has consequences on scientific practice, and in particular on the how experimental resources and copora should be dealt with, which can be translated in what I would consider rules of scientific ethics (which are not confined to linguistics):

  • A corpus of data should be precisely sourced and documented with all relevant information regarding the context and means of its creation. It should, as much as technically feasible include the initial raw data.

  • A corpus should not be tampered with for any reason, by anyone including particularly its creator.

  • However, it is naturally permissible to "correct", reprocess, annote, document or otherwise modify a corpus as long as all modifications are reproducibles by anyone, thus based on processors or separate data files that can be reapplied to the original files to get the modified version of corpus.

In other words, any phenomenon observed when processing a corpus should be traceable by anyone, for example to determine whether it comes from the original data or from whatever modification was applied to the corpus. This includes correcting what may seem obvious errors.

As long as you are transparent regarding these rules, you are free to do as you choose, preferably justifying your choices on the basis of your intent or technical means. Others may disagree with you, but they can then do so on an objective basis, and possibly do competing work on that same basis.

Then there is the issue of defining: what is an error? I do expect they can be categorized to some extent, though an alleged mistake could possibly belong to several categories. What you want to do may depend on these categories and unless you identifies a clear misconception in the rules you have been using, I would not suggest you modify them to account for what you consider errors.

Errors are inherent to any form of information exchange and linguistic information is no exception. Whenever possible, systematic (mechanized) error handling, should thus be a normal component of any natural language processing system. It has its own importance from a technological or scientific point of view, even if it may have intrinsic limitations.

To consider a specific example, I was involved in a project that did morphology based part-of-speech recognition to create lexicons. The system worked pretty well on most words. But there was a statistical dimension that caused chance errors to occur. Cases that were considered uncertain were reviewed by hand. But whatever modifications were made (by hand) to the raw output of the system were collected on a file, so that they were traceable, and could also be replayed or compared with the output of later version of the system. The idea is also that one should not correct twice the same error.

My philosophy is that anything done by hand should be minimized, traceable, and replayable in other contexts.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.