4

What language pairs are under-served by current resources — like human translators, bilingual dictionaries, and parallel corpuses — relative to their linguistic importance, economic potential, or human necessity?

There is research on using machine learning techniques to construct bilingual dictionaries from monolingual corpuses using representations like word embedding models. These techniques seem flexible enough that one could use them to relate arbitrary language pairs. The research would maybe have a bigger impact if it targeted under-served language pairs first.

Now, linguistic importance drives linguists to build resources. Economic potential motivates private industry. Cultural similarity or geographical proximity produces more humans who know related pairs of languages. But I suspect these effects aren't uniform. I'm interested in insight from linguists as to languages for which the tools don't measure up to the need.

Improve this question
6
  • 5
    There are around 7000 languages. Almost all of them are entirely undeserved by machine learning tools.
    – curiousdannii ♦
    May 1, 2018 at 14:00
  • 2
    @curiousdannii But he explicitly wrote "relative to their linguistic importance, economic potential, or human necessity".
    – Adam Bittlingmayer
    May 2, 2018 at 18:15
  • 1
    @curiousdannii Also, he asked for pairs, so ~7000^2.
    – Adam Bittlingmayer
    May 2, 2018 at 18:15
  • @curiousdannii I'm not asking which pairs are underserved just "by machine learning tools", I'm asking which are underserved overall, by traditional human resource in addition to by machine learning tools. And, A. M. Bittlingmayer is absolutely right, I intentionally qualified the question "relative to their linguistic importance, economic potential, or human necessity".
    – Jim DeLaHunt
    May 2, 2018 at 18:32
  • 1
    "Economic potential" is also very much a question about perspective. In the immediate term, this suggests Chinese and Hindi, but longer term, African and smaller Asian languages. If you add "who is going to be paying for this", working on some of the smaller EU languages can probably elicit good EU funding.
    – tripleee
    May 3, 2018 at 7:06

1 Answer 1

Reset to default
6

This question is very important and possible to answer empirically, however, words and concepts do not map 1:1 across languages so the mentioned assumption that bilingual dictionaries will have a great impact is speculative.

Relative to what we might expect based on economic factors and inherent difficulty, machine translation quality lags for:

English to x

Major eng orgs tend to notice and fix x to English quality issues. Funding and datasets from defence orgs like DARPA are also biased towards solving x to English.

User demand is actually in the other direction, because % of content in English (~1/2) is greater than the % of users who speak English (~1/4, and ~1/8 native).

non-English pairs

Pairs like Chinese-Spanish or French-Hindi have almost no support at all, despite being composed of major languages. The relative exception is Chinese-Russian, and French-German, depending on whether we talk about products or research results.

closely-related pairs

From a purely technical perspective it would be possible to achieve very very high quality on pairs like Portuguese-Spanish or French-Italian, however currently in all major production systems these pairs are via English, not direct. Even Hindi-Urdu is via English.

Less closely related pairs like Korean-Japanese or Chinese-Vietnamese would also benefit from not going via English.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.