I have a very large corpus of historical news papers (17th-20th cent.). The word count is about 20 bln. It's raw OCR-ed data in txt-files of about 150 GB. One newspaper issue per file (some 12 mln files).
Here are some further parameters:
- Some of the documents are completely jumbled up (not readable by humans).
- A lot of documents are alright (mostly readable by humans), but have still frequent problems. The classic among the errors is the 'long s' that is (inconsistently) confused with the "f" character.
- Often the respective newspaper layouts were not considered for the txt-output of the OCR process.
- Access to the (relatively good-quality) PDF-scans is possible but comes with certain challenges (that could be overcome if absolutely necessary).
- The language is a well-researched Germanic language but not English (I would rather not go into details).
- The corpus is diachronically skewed, i.e. one quarter of the data is from up 1870, then half the data is from 1870 to 1920, the last quarter is post-1920.
How would I best go about cleaning this kind of data?
- I don't have the illusion to be able fix everything.
- It's more a matter of weeding out the parts with bad OCR, i.e. maybe anything beyond a certain degree of badness.
- I would like to be able to use the corpus for further NLP analysis after clean-up (tokenising, tagging, parsing, ... but that's a whole 'nother story).
My musings as to how to tackle the issue:
- A thought I had was to use a dictionary and see how to what extent ocr-words can found in a dictionary. Using healthy text files, I could define a benchmark for OCR-quality.
Any help and pointers in the right direction would be greatly appreciated. Thank you in advance.