How to best clean a large historical corpus ridden with OCR errors

Question

Overview:

I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some 12 mln files).

Here are some further parameters:

• Some of the documents are completely jumbled up (not readable by humans).
• A lot of documents are alright (mostly readable by humans), but have still frequent problems. The classic among the errors is the 'long s' that is (inconsistently) confused with the "f" character.
• Often the respective newspaper layouts were not considered for the txt-output of the OCR process.
• Access to the (relatively good-quality) PDF-scans is possible but comes with certain challenges (that could be overcome if absolutely necessary).
• The language is a well-researched Germanic language but not English (I would rather not go into details).
• The corpus is diachronically skewed, i.e. one quarter of the data is from up 1870, then half the data is from 1870 to 1920, the last quarter is post-1920.

How would I best go about cleaning this kind of data?

• I don't have the illusion to be able fix everything.
• It's more a matter of weeding out the parts with bad OCR, i.e. maybe anything beyond a certain degree of badness.
• I would like to be able to use the corpus for further NLP analysis after clean-up (tokenising, tagging, parsing, ... but that's a whole 'nother story).

My musings as to how to tackle the issue:

• A thought I had was to use a dictionary and see how to what extent ocr-words can found in a dictionary. Using healthy text files, I could define a benchmark for OCR-quality.

Any help and pointers in the right direction would be greatly appreciated. Thank you in advance.

Answer 1

For English there exists a list of Basic OCR corrections by Ted Underwood and Loretta Auvil. In the linked blog they also explain how they generated that list of corrections by simulating typical errors automatically.

We improved on that for the Royal Society Corpus and our scripts to do that are available for download here. Our approach is tuned for precision, not for recall or f-score. Our improvements come essentially from a concentration on frequent function words and prefixes and suffixes.

A method for automated weeding of bad parts from the text base is to run language detection on rather small chunks of the text. Take out every chunk that is not recognised as in the correct language.

EDIT: Here is a reference where language detection is used as one criterion to clean up a corpus before syntax parsing: Krielke et al. (2022), Tracing Syntactic Change in the Scientific Genre: Two Universal Dependency-parsed Diachronic Corpora of Scientific English and German, LREC 2022

Answer 2

ModelFront is made for predicting translation risk. But you can abuse it for a monolingual scenario like this.

(The main reason people do that is because they're about to translate it into 100 languages, and would rather fix the errors just once.)

There are two ways to try to do what you want to do.

1. Just pick a dummy target language, like English

Let's say the language is Yiddish. Translate the file from Yiddish to English with a machine translation option that supports it. It can be just a .txt file.

...
דאָס איז אַ פּראָבע.
...

2. Translate to and from the same language*

Create a parallel file, where the "translation" of every segment is just a copy of the segment itself. (They should be separated by a tab and the file should end with .tsv.)

...
דאָס איז אַ פּראָבע.    דאָס איז אַ פּראָבע.
...

Then use the other und option and with Yiddish as the target language.

The latter approach is slightly cheaper, since you'll save on the machine translation.

---

Note that this will work best if the document or corpus is segmented into sentences - one sentence per line. Otherwise, the scores won't mean much.

Why does it work? The main factors in a ModelFront risk score are roughly:

1. the source-side risk, i.e. the quality of the original text
2. the target-side risk, i.e. the quality of the translated text
3. the translation risk, i.e. whether the target is actually a translation of the source?

With the latter approach, 3 is not a risk, and 1 and 2 are nearly the same, so it effectively reduces it to a measure of text quality.

---

Full disclosure: I'm a co-founder of ModelFront. This obviously isn't the main or intended use of the product. If you have a few examples of bad OCR, we could probably use those to improve the system for you and your scenario.