Efficient linguistic algorithms for detecting language of a website?

Question

Some browser addons and web-services for website/dictionary translation sometimes offer a "automatic-language-detection" feature. This works more or less in my experience.

There is probably a variety of different possible algorithms, like indexing some words of a website and compare them with different dictionaries. Is this the way most of those software do language-detection, comparing 5-10 words of the website with many dictionaries or are there smarter and faster algorithms? For example, languages with romanic letters. What would be a very efficient and reliable criterion to derive the language without the need of many big dictionaries, so it might fit into a small app.

The wiki article for language detection mentions functions words as a older method. I would guess, words used after/before punctuation marks might be even more effective for most european languages? Especially on multi-language websites where distinct function words might occur in quoted text/link names?

Answer 1

A popular approach for language identification is to look at character n-grams: consecutive sequences of n characters from the text, and compare the resulting distribution with frequencies drawn from larger corpora of text from each of the candidate languages.

Now, where do the empirical counts come from? Any representative text from the language will do. The University of Leipzig corpus consists of spidered web text for a large number of languages.

The intuition is that particular character sequences are characteristic of particular languages: for example, Dutch text is likely to have ij in it, Italian text should have a decent frequency of ità, and so on. The distribution of character n-grams is like the signature of a language.

This should be more robust than a dictionary-based method, particularly because a lot of short words are, just by chance, possibly going to occur in more than one language (e.g. in, al, do).

For certain languages which have scripts that are not shared with other languages (e.g. Chinese, although some of its topolects have written forms too), simple heuristics should suffice. Japanese and Chinese can be distinguished by the presence or absence of kana. On the other hand, if the text is romanised, then the character-based method should work.

This paper has a basic literature review and concentrates on the harder problem of language identification where the text is very short.

Answer 2

Not efficient but readily available : https://github.com/jaukia/cld-js - Compact Language Detector in Javascript. Detect the language of any piece of text.

Answer 3

Chrome/Chromium uses the open-source Compact Language Detector. You could use it too.

It does not work perfectly, as it is mainly designed to be compact (small, and fully local - no calls to the network), but Google and Chrome are behind it, and it is improving. It even runs on Chrome for Android.

So for your needs, it may be a good fit.

Also, in my experience, the right language detection behaviour depends highly on your actual application. Perhaps you can tell us how you intend to use it? (Will the strings you pass be pages/sentences/queries/words? Will it be clean data? Will there be mixed language input? Do you care more about accuracy or recall?)