How good is a round-trip translation as a machine translation quality evaluation technique?

Question

I recently had a text translated into Icelandic. When I machine translate that human translation back to English, it came out as quite awkward English. However, when I machine translate the machine translation back to English, it came out sounding perfect.

Is that just because it has high fidelity going both ways, even though it is wrong? Or should I suspect the human translator's version is wrong and he does not actually speak Icelandic well?

---

The original English text:

I felt it was a reasonable assumption that the developers would know about this method.

The human translation into Icelandic:

Mér fannst eðlilegt að ganga út frá því að leikjahöfundarnir þekktu til aðferðarinnar.

The machine translation into Icelandic:

Ég fann að það var sanngjarnt forsendu að verktaki myndi vita um þessa aðferð.

---

*I used Google Translate for the machine translation.

Answer 1

A so-called round-trip translation is not a reliable signal that the translation worked well, for a few reasons.

Noisy parallel corpora are bi-directional.

Machine translation systems train on parallel corpora, which are often noisy. For example, suppose the following noise occurs in the training data:

en: "This is a rare sentence."  is: "Rænðöm rænðöm gíbberish."

Note that there is no sense of directionality here; the English-Icelandic system learns:

en: "This is a rare sentence." -> is: "Rænðöm rænðöm gíbberish."

while the Iceland-English system learns:

is: "Rænðöm rænðöm gíbberish." -> en: "This is a rare sentence." 

In this simplified example, the output of the round-trip translation - trans('is', 'en', trans('en', 'is', x)) - will likely be x. The two mistakes cancel each other out. This happens more often than it would if translation mistakes were simply random.

Even good translation is lossy.
A translation, and even a language itself, is lossy. Distinctions of meaning, gender, tu/vous, tense and aspect are not represented equivalently or at all in all languages. Ambiguities are also not necessarily represented.

Let us consider an example, with English as the intermediate language to make it clear:

es: "La profesora Le está esperando."

We can legitimately translate this into English as:

en: "The teacher is waiting for you."

[Note: We lose the gender of the teacher, whether it is a schoolteacher or university professor, and the formality of you, and any choices about word order.]
Now we translate back into Spanish:

es: "Te espera el maestro."

That round-trip translation very different than the original, although the translation in both steps was correct. This dynamic applies to both human and machine translations.

Machine translation is not perfect.
The lack of readability of the machine translation of Icelandic text written by a human is possibly a signal that the human did not use that machine translation system but did, in fact, translate by hand in some way. In fact, good idiomatic translations are more difficult for machine or human translators.

The last dynamic applies to evaluating a human translation by machine translating it into some language you know.

Answer 2

To continue the experiment, and in support of the answer already given by Adam, I submit the following:

Google Translate now (2023) produces this translation:

Mér fannst það vera hæfileg forsenda að verktakarnir myndu vita um þessa aðferð.

This is a better translation than the 2016 one in the sense that it no longer contains any grammatical errors (the earlier one has at least two). The only other major change is that reasonable is now translated as “hæfileg”, a word that could possibly work in a context like reasonable amount but more commonly means something like ‘appropriate’ or ‘just right’.

The new translation can be used to illustrate two of the points made by Adam:

Even good translation is lossy.

• There is loss in the translation to Icelandic because reasonable is (almost unavoidably) translated by a word that only partly corresponds to it semantically. This in turn makes it more likely that any translation back into English will be incorrect, misleading or even unintelligible.

Machine translation is not perfect.

• The machine translation renders the developers as “verktakarnir“, a word that means ‘contractor’ and therefore can sometimes refer to, for example, a real estate developer, but is never used in the meaning ‘game developer’.