Take the 2-minute tour ×
Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It's 100% free, no registration required.

I will be attempting to solve the problem of automatic language identification/detection (and later translation), and I'm in a need of free digital dictionaries or corpora.

I'm looking for dictionaries for English, German and Slovenian language. Each dictionary must have words translated to all other mentioned languages. I only need word--word/"list of words" kind of dictionary, I don't need the definitions, nor examples. Preferably, they should contain word frequencies/probabilities, or at least be sorted from most used to least.

Since the above request seems a bit unreasonable, I'm completely fine with large corpora (50GB+ of plain text is preferred; but at least 10GB is fine too) for each language. Preferably noiseless, which means that the German language shouldn't contain (many) English words.

share|improve this question
add comment

3 Answers

up vote 5 down vote accepted

The largest parallel corpus I know of is the Europarl Parallel Corpus. It contains sentence-aligned bi-texts, including German-English and Slovene-English. However, it doesn't come close to the quantities of text you are asking for (German ~ 47 million words, Slovenian close to 13 million words). (Also see the EU translation memory).

I doubt huge corpora exist for Slovenian at all, especially if you need parallel corpora.

Here are some more resources for locating corpora:

Regarding English loanwords, it's important to realize loanwords are an integral part of language. Especially in domains like technology English loans are unavoidable. That said, different kinds of texts will likely contain different proportions of loanwords (and code switching), but I think this is something you will have to establish by analysing the available material.

share|improve this answer
    
Nice answer, thanks! I was thinking of using Wikipedia as the corpus for all three languages, but wasn't sure about loanwords. It seems I should stop fighting the natural evolution of languanges. –  Howie Jan 28 '13 at 7:50
add comment

You may be interested in this comparison of 3 language detection libraries.

In particular, language-detection is trained on Wikipedia and achieves 99.22% accuracy on single lines from Europarl corpus, so using Wikipedia shouldn't be a problem for accuracy.

share|improve this answer
add comment

A word--word/"list of words" kind of dictionary is not feasible, because in each language many words have multiple meanings. For many words meanings overlap. For example:

Slovenian     English     
-------------------------------------------
zastonj       in vain     (without success)
zastonj       free        (not costing any money)
svoboden      free        (not slave)
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.