I will be attempting to solve the problem of automatic language identification/detection (and later translation), and I'm in a need of free digital dictionaries or corpora.

I'm looking for dictionaries for English, German and Slovenian language. Each dictionary must have words translated to all other mentioned languages. I only need word--word/"list of words" kind of dictionary, I don't need the definitions, nor examples. Preferably, they should contain word frequencies/probabilities, or at least be sorted from most used to least.

Since the above request seems a bit unreasonable, I'm completely fine with large corpora (50GB+ of plain text is preferred; but at least 10GB is fine too) for each language. Preferably noiseless, which means that the German language shouldn't contain (many) English words.

3 Answers 3


The largest parallel corpus I know of is the Europarl Parallel Corpus. It contains sentence-aligned bi-texts, including German-English and Slovene-English. However, it doesn't come close to the quantities of text you are asking for (German ~ 47 million words, Slovenian close to 13 million words). (Also see the EU translation memory).

I doubt huge corpora exist for Slovenian at all, especially if you need parallel corpora.

Here are some more resources for locating corpora:

Regarding English loanwords, it's important to realize loanwords are an integral part of language. Especially in domains like technology English loans are unavoidable. That said, different kinds of texts will likely contain different proportions of loanwords (and code switching), but I think this is something you will have to establish by analysing the available material.

  • Nice answer, thanks! I was thinking of using Wikipedia as the corpus for all three languages, but wasn't sure about loanwords. It seems I should stop fighting the natural evolution of languanges.
    – Howie
    Jan 28, 2013 at 7:50

You may be interested in this comparison of 3 language detection libraries.

In particular, language-detection is trained on Wikipedia and achieves 99.22% accuracy on single lines from Europarl corpus, so using Wikipedia shouldn't be a problem for accuracy.


A word--word/"list of words" kind of dictionary is not feasible, because in each language many words have multiple meanings. For many words meanings overlap. For example:

Slovenian     English     
zastonj       in vain     (without success)
zastonj       free        (not costing any money)
svoboden      free        (not slave)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.