Looking for bi-/tri-lingual dictionaries or corpora

Question

I will be attempting to solve the problem of automatic language identification/detection (and later translation), and I'm in a need of free digital dictionaries or corpora.

I'm looking for dictionaries for English, German and Slovenian language. Each dictionary must have words translated to all other mentioned languages. I only need word--word/"list of words" kind of dictionary, I don't need the definitions, nor examples. Preferably, they should contain word frequencies/probabilities, or at least be sorted from most used to least.

Since the above request seems a bit unreasonable, I'm completely fine with large corpora (50GB+ of plain text is preferred; but at least 10GB is fine too) for each language. Preferably noiseless, which means that the German language shouldn't contain (many) English words.

Answer 1

The largest parallel corpus I know of is the Europarl Parallel Corpus. It contains sentence-aligned bi-texts, including German-English and Slovene-English. However, it doesn't come close to the quantities of text you are asking for (German ~ 47 million words, Slovenian close to 13 million words). (Also see the EU translation memory).

I doubt huge corpora exist for Slovenian at all, especially if you need parallel corpora.

Here are some more resources for locating corpora:

• http://opus.lingfil.uu.se (includes Slovenian corpora)
• http://www.ldc.upenn.edu/Catalog/
• http://www.lt-world.org/kb/resources-and-tools/language-data

Regarding English loanwords, it's important to realize loanwords are an integral part of language. Especially in domains like technology English loans are unavoidable. That said, different kinds of texts will likely contain different proportions of loanwords (and code switching), but I think this is something you will have to establish by analysing the available material.

Answer 2

You may be interested in this comparison of 3 language detection libraries.

In particular, language-detection is trained on Wikipedia and achieves 99.22% accuracy on single lines from Europarl corpus, so using Wikipedia shouldn't be a problem for accuracy.