4

Are there any statistic on how accuracy the automated (program/machine) transition from one language to another language of the same language family?

For example from turkey to azerbaijan, uzbek, turkmen etc (turkic langs) They all have same logic (suffix addition order etc).

  • I suspect "same family" as a binary distinction isn't very useful. German, Spanish, and Hindi are in the same family, but a lot more distantly related than, say, German, Swedish, and Dutch. (Not to mention that sometimes two closely-related languages can have major differences in grammar, while sometimes a broad family with dozens of languages can be grammatically very consistent.) – abarnert Jan 14 '19 at 21:49
  • Also, given how much machine translation work involves English, I suspect you'd be a lot better off looking for comparisons between, say, English/German, English/Hindi, and English/Turkish than, between say, Turkish/Azerbaijani, Turkish/Kipchak, and Turkish/something-not-Turkic-at-all. Would those English-based statistics be acceptable? – abarnert Jan 14 '19 at 21:53
  • Anyway, I know Google has published some results on the accuracy of GNMT and its predecessor. For example, this blog post says that (where the metric is apparently native speakers evaluating translations subjectively on a scale of 0 to 6) English/French scores 5.43, while English/Chinese scores only 4.3, although they don't give the source for that. – abarnert Jan 14 '19 at 21:58
  • @abarnert The languages in question are much more closely related than IE languages, a better comparison would be Germanic languages. – Adam Bittlingmayer Jan 15 '19 at 8:01
  • 1
    @Probably Maybe that's because google translate use direct translation between Czech and Slovak. But there are no official documents that indicate which pairs use direct translation. – UserKa Jan 16 '19 at 7:23
5

You're right to suspect that the accuracy can potentially be very good, but, in practice, unfortunately, as of 2019, most of the major systems - those from Google, Microsoft, Baidu, Yandex, Facebook, Amazon, DeepL and so on - use bridging via English for almost all pairs, even closely related ones.

The reason for this is simply pragmatic. Those systems support 100+ languages, and thus there are 10000+ pairs. Most of those pairs, like Tamil-Basque, are not in high demand, and there is not much training data for them anyway. That's even true for pairs of related languages like Romanian-Galician.

Engineering-wise, even if one could create data for, train and evaluate all those pairs, it is also just a lot of effort to deploy and maintain 10000+ systems in production at scale, even with NMT - end-to-end models.

So, incredibly, even very closely related major languages are not translated directly, but via English. You can easily test this, for example for Spanish-Portuguese, which are almost as close as Turkish and Azerbaijani, and have much much more training data.

enter image description here

In this example, He was killed by a bat is ambiguous in English - it's not clear if the flying mouse or the wooden stick is meant.

(If you use machine translation API and are interested catching such casualties of bridging and other errors, you can try machine translation risk prediction like ModelFront.)

Turkish-Azerbaijani is actually one of the lucky pairs that does have a direct system in at least one direction on Google Translate. Let's test it. bat doesn't work as well in Turkish since they basically just say baseball stick, so we can use the T-V distinction as an example of something that English cannot represent.

enter image description here enter image description here

We've confirmed that a translation via English would probably lose it.

enter image description here

And, yes, Turkish-Azerbaijani still works as it should. So we can conclude it's direct. (Although note that the bridging could pass along some hints, so we should test this really well before making strong conclusions.)

One reason for this approach is that it would be hard to get the accuracy for English-Azerbaijani. There is simply much more English-Turkish and Turkish-Azerbaijani data.

So, about the accuracy, well it could potentially be one of the best pairs, but it needs more data and more work than has been put into it. As far as I know, English-Portuguese outperforms for example English-Dutch and even English-Frisian on all major engines, even though Frisian is probably the language closest to English supported by any major system.

The blocker here is really that most societies are not home to a major technology company. The major systems are made in the US, China and Russia, without exception. The rest of the world just does not produce much. The closest candidates are Systran, which is not very competitive in recent decades, and DeepL, which only covers European languages, and still focuses on pairs with English, English being the lingua franca of Europe. Turkey and Azerbaijan inflicted braindrain on themselves throughout the 20th century, and show no signs of stopping, so my bet is that Yandex will be the first to build direct systems for more obscure Turkic pairs.

For more context see my answers to Which languages are Google Translate best at translating? and Which two languages is machine translation worst at translating between?.

| improve this answer | |
  • Does Baidu bridge everything through English as much as the western companies? (I can imagine that when they started, and were trying to ramp up to near-Google-equivalency as quickly as possible, they used whatever corpus data was available, which was mostly English/___ translations. But they've had a few years since then to build up and train on Mandarin/___. And it seems like that would give them a competitive advantage against Google/etc., and also be something the government would encourage/facilitate.) – abarnert Jan 15 '19 at 8:25
  • How can I find out which language using bridge English or not. Are there any documents? How did you find out that Turkish / Azerbaijani has direct system? Or is this just your guess based on your example? – UserKa Jan 15 '19 at 8:41
  • And another question (I don't think that I should create new question topic for this) Do you know any translation service that uses one of the Turkic languages as a bridge (pivot) language? – UserKa Jan 15 '19 at 9:42
  • @abarnert No, Yandex, Baidu, Youdao etc may bridge through Russian or Chinese, and of course they do x-Russian and x-Chinese directly, but that has the same issue. Just like Google, Facebook, Microsoft and Amazon, 90% of their pairs are bridged not direct. – Adam Bittlingmayer Jan 15 '19 at 10:53
  • @UserKa Look at my answer, I showed you how can you find out that a system is bridging via English (Spanish-Portuguese). It's hard to prove that a system is not bridging via English, but if you test enough and see no signs that it is, which are usually obvious, then you can start to assume it's not. – Adam Bittlingmayer Jan 15 '19 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.