What is the state of the art for diacritic restoration?

Question

Diacritic restoration is a term for guessing the original form of a text, after it's been converted to ASCII in a lossy way. In Lingála, for example, writing in ASCII merges two pairs of vowels and removes all tone markings: the words mɔ́tɔ "fire" and moto "person" aren't distinguished at all. (It doesn't always involve diacritics, as with the vowel letters here, but the name has stuck.)

The only diacritic restoration programs I've found for Lingála use a very simple statistical analysis: ASCII "moto" corresponds to moto more often than mɔ́tɔ, therefore all instances of "moto" are restored as moto. But this isn't particularly satisfying: tones and vowel qualities separate thousands of minimal pairs in Lingála (for instance, certain verb tenses are distinguished only by tone).

Are there systems out there, for Lingála or other languages, which can take context into account? What are the best models currently used? In other words, what's the state of the art here?

Answer 1

The basic approach described in https://github.com/atpaino/deep-text-corrector and https://yerevann.github.io/2016/09/09/automatic-transliteration-with-lstm/ is still the state of art.

The recipe is:

1. Generate data
   a. Build a corpus of unlabelled raw text in the target language
   b. Convert each line to the source language using hand-built rules
   c. Flip the sides, so that the source language just output is now on the input side
2. Train a model
   Train a model to convert from source language to target language

The above assumes that 1b is possible - but it need not be perfect, it is fine to generate source never seen in the wild - and that context helps disambiguate.

The main advances from 2016 to 2018 are in the architecture used in training the model. The state of the art would be some flavour of sequence-to-sequence - implementations like Fairseq or Sockeye - with attention, byte-pair encodings, reversed source input...

Beyond state of the art would be transfer learning - training a single model that does the conversion for multiple languages - and various approaches with target-side language models - boosting accuracy with unlabelled raw text in the target language. See https://deepchar.github.io/