3

I am a machine learning developer who is currently working with natural language processing.

Today, I was thinking about language identification. I have a couple of ideas how to do so for the ~200 most common languages.

Now I'm thinking about how to design the programmers interface. I want a function

identify_language: string -> list of tuples (ISO 369-3, score)

My approach would be to make each score a probability. Say we have a text (of sufficient length; for the sake of discussion say at least 140 characters as counted by Twitter). And suppose it belongs to one language (e.g. no URLs, no codes; just natural language): Does it always belong to exactly one language?

Improve this question
4
  • But… returning a list of probabilities assumes that there can be several possible answers. What is your concern? Is your goal to try eliminating the list?
    – Be Brave Be Like Ukraine
    Aug 2, 2017 at 6:20
  • 1
    No. Each element of the list I return has a probability. Hence I could return a list of all languages, but except for two each have probability 0. Now the interesting part is normalization. Should all probabilities of the languages add up to 1 or could there be something like [(english, 0.8), (german, 0.7)] because the sentence might actually be a valid sentence in both languages?
    – Martin Thoma
    Aug 2, 2017 at 6:27
  • 3
    "1" is an abstraction. You normally have a neuronet that would return single-language probabilities. If you need normalizing it, you would do it manually, e.g. 0.8+0.7=1.5 and then get [(EN, 0.5333); (DE, 0.4667)], And yes, I think I can craft several cross-language homograph phrases that would be 80% valid in, say, Ukrainian, Serbian, and Russian at the same time.
    – Be Brave Be Like Ukraine
    Aug 2, 2017 at 6:55
  • 2
    I guess languages as closely related as Bosnian, Croatian, Montenegrin and Serbian can have identical sentences
    – Frédéric Grosshans
    Aug 3, 2017 at 12:12

3 Answers 3

Reset to default
3

If being the official written standardized form of the official language of an independent country is enough to be a “language”, the answer is yes. The difference between languages and dialects is often tricky and politically loaded, and in some cases, languages are very closely related.

I guess the best case showing this is the pluricentric Serbo-Croatian language, which, after the break-up of Yugoslavia, now corresponds to four official national languages (Bosnian, Croatian, Montenegrin, and Serbian). According to the Wikipedia page comparing these languages, the 1st article of the Univerwal Declaration of Human Rights (UDHR) can be translated as follows in Bosnian, Croatian and Serbian:

Sva ljudska bića rađaju se slobodna i jednaka u dostojanstvu i pravima. Ona su obdarena razumom i sviješću i treba da jedno prema drugome postupaju u duhu bratstva.

Since this sample is 164 characters long according to Twitter, it answers your question as specified.

Another set of languages I would look for possible confusions would be the Nordic languages, specifically Danish and Norwegian Bokmål. But most generally, if you want to automatically find naturally occuring similar strings of text for different languages, I would look at this set of UDHR translations.

Of course, if you are concerned about sentences specifically crafted to be ambiguous, I would bet it is possible to construct identical sentences in unrelated (or weakly related) languages, with different meanings.

Improve this answer
3

Yes.

There are sentences that are simply exactly the same in similar languages.

"No." or "Gol!" or Trump! would be some extreme examples, but between languages like Portuguese and Galician, Croatian and Bosnian or Danish and Swedish there is significant overlap in any parallel corpus. It is really then more a function of orthography.

Rarely there are also random or not so random collisions - "false friends" - where a phrase or sentence in one language means something completely different in another. "50ml Gift, 1 Roman & 1 Abort" means something very very different in German than in English.

The last example is not a sentence, but you will see plenty of not quite a sentence language in the Twitter data. See https://blog.twitter.com/engineering/en_us/a/2015/evaluating-language-identification-performance.html

You should also think what you want to do about a case like Pozor!, which is valid in almost all Slavic languages written in Latin, but could also be Russian позор given that on the internet people do write Russian in translit.

The input and output of language identification is application specific, we cannot speak of accuracy for this task in the abstract. In many applications, the probabilities should not add up to 1.0 as they are not mutually exclusive. Commonly for example the API client wants to know if content is in one of the user's languages.

Improve this answer
1

You'll also have sentences/examples containing chunks of one or more foreign languages, especially if you work with social media text.

You can have a tweet like: "Du KM classique au KM 2.0 : le renouveau du Knowledge Management à l’ère du digital", where you'll have a percentage of French and a percentage of English languages.

In this case I would rather add up at 1 and get as score the distribution of languages present in my example.

I guess it depends of what is your most frequent usecase.

Improve this answer
1
  • 2
    Loanwords and similar is not what I'm talking about. The sentence is sill clearly french.
    – Martin Thoma
    Aug 3, 2017 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.