I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some 12 mln files).

Here are some further parameters:

  • Some of the documents are completely jumbled up (not readable by humans).
  • A lot of documents are alright (mostly readable by humans), but have still frequent problems. The classic among the errors is the 'long s' that is (inconsistently) confused with the "f" character.
  • Often the respective newspaper layouts were not considered for the txt-output of the OCR process.
  • Access to the (relatively good-quality) PDF-scans is possible but comes with certain challenges (that could be overcome if absolutely necessary).
  • The language is a well-researched Germanic language but not English (I would rather not go into details).
  • The corpus is diachronically skewed, i.e. one quarter of the data is from up 1870, then half the data is from 1870 to 1920, the last quarter is post-1920.

How would I best go about cleaning this kind of data?

  • I don't have the illusion to be able fix everything.
  • It's more a matter of weeding out the parts with bad OCR, i.e. maybe anything beyond a certain degree of badness.
  • I would like to be able to use the corpus for further NLP analysis after clean-up (tokenising, tagging, parsing, ... but that's a whole 'nother story).

My musings as to how to tackle the issue:

  • A thought I had was to use a dictionary and see how to what extent ocr-words can found in a dictionary. Using healthy text files, I could define a benchmark for OCR-quality.

Any help and pointers in the right direction would be greatly appreciated. Thank you in advance.

  • How much of the data are you willing to discard if necessary? – Draconis Feb 5 at 14:49
  • @Draconis -- That depends on the quality of what's left. If the reduced corpus is impeccable, I would be very happy with 10-20% (discarding 80-90%). The corpus is diachronically skewed, i.e. one quarter of the data is from up 1870, about half the data is from 1870 to 1920, the last quarter is post-1920. Presumably the worst OCR is from earlier periods. (I'll add this note on diachronic structure in the post above in a second.) – Mat Feb 5 at 15:39
  • I would consider constructing some sort of dictionary of frequently misread words or letter sequences and their probable translations. This could be used purely on words that are not found in a regular dictionary, or, with info on the degree of garbling for the document being studied, replacement could be done on a probabilistic basis. – Hot Licks Feb 5 at 23:07

For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical errors automatically.

We improved on that for the Royal Society Corpus and our scripts to do that are available for download here. Our approach is tuned for precision, not for recall or f-score. Our improvements come essentially from a concentration on frequent function words and prefixes and suffixes.

A method for automated weeding of bad parts from the text base is to run language detection on rather small chunks of the text. Take out every chunk that is not recognised as in the correct language.

  • That is a fantastic resource (the OCR clean-up for the RSC). I made an edit to the post indicating that it isn't (older forms of) English I'm working on. – Mat Feb 5 at 15:50

ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this.

(The main reason people do that is because they're about to translate it into 100 languages, and would rather fix the errors just once.)

There are two ways to try to do what you want to do.

1. Just pick a dummy target language, like English

Let's say the language is Yiddish. Translate the file from Yiddish to English with a machine translation option that supports it. It can be just a .txt file.

דאָס איז אַ פּראָבע.

enter image description here

2. Translate to and from the same language*

Create a parallel file, where the "translation" of every segment is just a copy of the segment itself. (They should be separated by a tab and the file should end with .tsv.)

דאָס איז אַ פּראָבע.    דאָס איז אַ פּראָבע.

Then use the other und option and with Yiddish as the target language.

enter image description here

The latter approach is slightly cheaper, since you'll save on the machine translation.

Note that this will work best if the document or corpus is segmented into sentences - one sentence per line. Otherwise, the scores won't mean much.

Why does it work? The main factors in a ModelFront risk score are roughly:

  1. the source-side risk, i.e. the quality of the original text
  2. the target-side risk, i.e. the quality of the translated text
  3. the translation risk, i.e. whether the target is actually a translation of the source?

With the latter approach, 3 is not a risk, and 1 and 2 are nearly the same, so it effectively reduces it to a measure of text quality.

Full disclosure: I'm a co-founder of ModelFront. This obviously isn't the main or intended use of the product. If you have a few examples of bad OCR, we could probably use those to improve the system for you and your scenario.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.