2

In recognising the language of a text, spoken or written, have current algorithms or implementations incorporated ISO 639 codes and/or other standards?

For example, I'd like to decide the language of a text, but map the text directly to an ISO 639 code. Is it currently possible?

If yes, how is this implemented? I can guess that this would require that either ISO 639 specify ways on how to decide whether a text belongs to a language code, or the registrars of ISO 639 keep adequate samples of each language code. However, from my current understanding (I may be wrong because I have no direct access to the standard), these aren’t true.

Or are there others ISO or other organisations' standards that deal with this issue directly?

  • 3
    ISO 639 categorisation is the least difficult and least important part of language recognition. Even if some tool outputed the language in some other code it would be relatively trivial to map it to ISO 639 codes. – curiousdannii Aug 4 '19 at 12:17
  • Are you asking if it's possible to make a language classifier with outputs that look like "eng" or "en" instead of "English"? If so, the answer is yes. Are you asking if the ISO provided a classification algorithm? If so, I'm pretty sure the answer is no. Are you asking if there are any classifiers that have full coverage of the ISO-639 languages and divides up the space the languages the same way? If so, then I don't know. Is there a particular classifier you tried that differs from ISO-639 in a way that doesn't work for you? – Jetpack Aug 7 '19 at 2:09
2

There's CLD3, which is the model used in the Google Chrome browser. The training data and code is not available, but the inference code and a pre-trained model is open source.

There's Python bindings and Ruby bindings for the original C++ code

>>> cld3.get_language("This is a test")
LanguagePrediction(language='en', probability=0.99, is_reliable=True, proportion=1.0)

>>> cld3.get_frequent_languages("This piece of text is in English. Този текст е на Български.", 5)
[LanguagePrediction(language='bg', probability=0.92, is_reliable=True, proportion=0.58),
 LanguagePrediction(language='en', probability=0.99, is_reliable=True, proportion=0.41)]

Looks like it usually outputs ISO-639-1 (2 letter) codes, but can also output 3 letter codes or "unknown"

| improve this answer | |
0

I don't think so, because of lack of training data for many languages with an ISO 639 code.

While language recognition in general is really good and while it can do really fine grained distinctions as well, it is still a machine learning process that depends on training data. It can only recognize what it has seen before and will react unpredictable on languages missing from the training set.

Even with sufficient training data I am not sure whether a classification into several thousand categories is still working (though several dozen languages at once work pretty well).

| improve this answer | |
  • Just because you're might get low accuracy for the long tail of languages (and languages that are considered separate languages for political reasons) doesn't mean it's not possible. I don't think this answer is useful. – Boris Aug 6 '19 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.