Are there any statistics on the accuracy of automated (program/machine) translation from one language to another language of the same language family?
For example from Turkish to Azerbaijani, Uzbek, Turkmen etc. (Turkic langs). They all have the same logic (suffix addition order etc.).
Update for 2024: Not much has changed since 2022.
Update for 2022: Much has changed since 2019. I added updates at the bottom.
You're right to suspect that the accuracy between closely related languages can potentially be very good, but, in practice, unfortunately, as of 2019 2024, most of the major systems - those from Google, Microsoft, Baidu, Yandex, Facebook, Amazon, DeepL and so on - use bridging via English for almost all pairs, even closely related ones.
They don't translate directly between languages if one of the languages is not English.
The reason for this is simply pragmatic, more economic than linguistic. Those systems support 100+ languages, and thus there are 10000+ pairs. Most of those pairs, like Tamil-Basque, are not in high demand, and there is not much training data for them anyway. That's even true for pairs of related languages like Romanian-Galician.
Engineering-wise, even if one could create data for, train and evaluate all those pairs, it is also just a lot of effort to deploy and maintain 10000+ systems in production at scale, even with NMT - end-to-end models.
So, incredibly, even very closely related major languages are not translated directly, but via English. You can easily test this, for example for Spanish-Portuguese, which are almost as close as Turkish and Azerbaijani, and have much much more training data.
In this example, He was killed by a bat is ambiguous in English - it's not clear if the flying mouse or the wooden stick is meant.
Turkish-Azerbaijani is actually one of the lucky pairs that does have a direct system in at least one direction on Google Translate. Let's test it. bat doesn't work as well in Turkish since they basically just say baseball stick, so we can use the T-V distinction as an example of something that English cannot represent.
We've confirmed that a translation via English would probably lose it.
And, yes, Turkish-Azerbaijani still works as it should. So we can conclude it's direct. (Although note that the bridging could pass along some hints, so we should test this really well before making strong conclusions.)
One reason for this approach is that it would be hard to get the accuracy for English-Azerbaijani. There is simply much more English-Turkish and Turkish-Azerbaijani data.
So, about the accuracy, well it could potentially be one of the best pairs, but it needs more data and more work than has been put into it. As far as I know, English-Portuguese outperforms for example English-Dutch and even English-Frisian on all major engines, even though Frisian is probably the language closest to English supported by any major system.
The blocker here is really that most societies are not home to a major technology company. The major systems are made in the US, China and Russia, without exception. The rest of the world just does not produce much. The closest candidates are Systran, which is not very competitive in recent decades, and DeepL, which only covers European languages, and still focuses on pairs with English, English being the lingua franca of Europe. Turkey and Azerbaijan inflicted braindrain on themselves throughout the 20th century, and show no signs of stopping, so my bet is that Yandex will be the first to build direct systems for more obscure Turkic pairs.
For more context see my answers to Which languages are Google Translate best at translating? and Which two languages is machine translation worst at translating between?.
——
Looking back 3 years later, a few points need updating.
Russia is also inflicting brain drain on itself and the Yandex team.
DeepL, made in Germany, is now a major player and supports dozens of languages, including East Asian languages.
Massively multilingual models - that don’t require bridging - are moving towards production.
What Turkic languages are concerned, the rule-based open source machine translation system Apertium has direct translations between the Turkic languages implemented. It should outperform systems with English as an intermediate language, but I am not a specialist in this area and I don't know studies doing a quantitative comparison.
