2

Why is it that we need machine translation evaluation? If we have subjective evaluation measures why do we need automatic evaluation? As most of the automatic evaluation metrics are unable to capture linguistic features of languages (Particularly non english, free word order languages).

More over can one or more automatic evaluation metrics capture deeper linguistic errors that MTs most often make. I don't mean to be rude. I don't mean to offend anybody. It is just that I am new to MT and while reading various papers and books have lots and lots of curious queries.

Improve this question

1 Answer 1

Reset to default
6

Here's a link to the paper introducing BLEU, the most widely used automatic MT evaluation metric. This metric is problematic in many ways, but the paper does address the question you are asking:

http://dl.acm.org/citation.cfm?id=1073135

In brief, human evaluation of MT is too time consuming to be the sole method used. Quick progress in the field relies on developers being able to run experiments testing the effect of different approaches and quickly get a sense of whether those approaches are helping or hurting.

Improve this answer
1
  • also, BLEU and other metrics yield reproducible results, while subjective evaluation outcome may vary depending on the evaluators and the instructions they are given. If one were to compare two systems using human judgement, they would have to include the output of both systems in the experiment. And when someone else repeats that comparison, they cannot be sure to get the same result.
    – Qnan
    Sep 21, 2012 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.