6

I recently reread What's the difference between the various context dictionaries available for Spanish (e.g., Tatoeba, Reverso, Linguee, …)?. The accepted answer is excellent. But it got me wondering whether there's a scientific methodology for evaluating bilingual dictionaries, and if so, how the dictionaries listed in that question fare?

I'm not necessarily looking for a final result. It could be an algorithm that either a computer program or a group of volunteers could go through to make a "scientific" (in some sense) evaluation/comparison.

The answer could be focused on Spanish, Romance languages, or languages in general.

  • 1
    The short answer is, no there isn't. We can unscientifically evaluate a dictionary when we known that a given dictionary sucks, because the translations are wrong or they miss major meanings. But that isn't a "method". – user6726 Nov 13 '19 at 2:25
  • @user6726 - The truth is, what I was hoping to do was get a measure of how unreliable Reverso is compared to some other resources. I'm fed up with people asking about awful sentences they found on Reverso, and having to tell them again and again that the sentences are nonsense. – aparente001 Nov 13 '19 at 4:16
  • Dictionaries don't give you arbitrary sentences, they give you words. So your interest is broader, viz. comparison of translation programs. – user6726 Nov 13 '19 at 16:15
  • @user6726 - I guess it depends where you draw the line. Linguee and Reverso do include definitions; some dictionaries do include examples of usage, some more than others. – aparente001 Nov 13 '19 at 16:18
  • The answer to the question you linked is wrong on a few major points. – Adam Bittlingmayer Nov 14 '19 at 19:26
6

That answer on Spanish SE is misleading on key points - "neural networks" have nothing to do with dictionaries.

Let's step back and imagine that we are tasked with creating bilingual dictionaries for many language pairs. To start, we have human-compiled ones, either from processing Wiktionary entries' Translations sections, or purchased from companies like Merriam-Webster or Langenscheidt.

What will be the challenges?

Quality

This is the obvious one. There will be garbage in there. Sometimes it's technically correct, but probably not what the user is looking for.

enter image description here As far as I know, plenty of native English speakers do not even know this sense of shanghai. Here you see also that the back-translation is a strong hint to the user that this may not be the desired translation.

Using quality estimation techniques, we can identify these for human review. Back in the day, we had the phrase tables. Now there is ModelFront.

Completeness - breadth

If we take the top n words - from our actual user queries, let's say - how many of them even have an entry? What about multi-word phrases?

Completeness - depth

And of the entries, how many of them actually have all the major possible translations of the word?

So, in that Shanghai example above, the translation Shanghái is missing.

Relevance

When there are multiple translations - and for the most common words, there often are - are they ordered correctly?

Frequency in some corpus is one metric, but it may not be what users want. It can also feel wrong in cases where a figurative sense or named entity is currently more popular than the original sense, so what should be the top translation for net or Java?

Special cases

Single-word translations are tricky, because they can be function words. How would you translate English the into German? Into Russian? Maybe it's better to have some note about Russian not using articles?

Named entities are another challenge. The ModelFront base model actually mark the literal translations as risky, because they are so often not what the client applications want.

enter image description here

Should dictionary results include named entities? Should translation always take the literal interpretation? In these cases, the UX matters, but for an application like Google Translate, arguably there should be multiple results, which is in some sense what dictionary results provide.

Searchability

Ideally, if you search for some morphological variant of a word, you should get the dictionary entry for the lemma. It could also include variants with clitics, like hablándole, or as parts of compounds. Defining what a word is is not simple, especially across languages. And of course, if the user doesn't use proper casing, or doesn't write the accent marks, the resulting entry should still be found, via fuzzy matching.

But fuzzy matching creates another challenge, because now a single query could be mapped to multiple source-side lemmata.

One way to deal with this is to prefer exact match. So if the user searches munchen or muenchen or Munchen or Muenchen or MUNCHEN..., we assume it was München. But if the user searches schon, we assume it was for schon because such an entry exists, and do not show results for schön.

But even exact match is tricky, it can end up hiding the better result. Surely some fraction of users searching schon actually meant schön. And lowercase English shanghai is technically an exact match for the verb, but should we not show the entry for the noun? Spanish habla is an exact match for the noun, but should we not show the entry for the verb?


Not sure if that counts as scientific, but it's how it happens in the real world. Overall these are really UX questions, application-specific UX questions. Until today, the UX of the applications in question suffer greatly from the fact that they do not know or do not consider whether the source or the target language is the one you know better. And any evaluation should probably also consider that. Stateless open-domain all-purpose machine translation is tough because you don't know who the user is and what the user wants.

There is a lot to appreciate about the Linguee approach and Linguee's initial focus on translation dictionaries over sentence translation, for users who are fluent in both languages.

Full disclosure: In my past life as a software engineer on Google Translate, I helped out with the project to improve Google Translate's dictionary results, under the direction of Keith Stevens and John DeNero, now CTO of Lilt. We improved it by orders of magnitude, but as you see, there is plenty still left to improve. I'm now a technical co-founder at ModelFront, a company dedicated to catching bad translations.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.