Are Word Frequencies Cross-Lingual?

Question

If someone tried to invent a code where they simply replaced every English word with another word, the code could be cracked (given a large enough sample) by comparing frequencies of English words to the frequencies of the code words. For example, since "the" is the most common English word and if a code-breaker were able to figure out that "dragef" is the most common code word, the code-breaker could know that "dragef" means "the."

The frequencies of each word in the code language will be the same as the frequencies of their corresponding words in English. So, I wonder, is the same true for other real languages? I realize that, because a word from one language rarely has an exact equivalent in another language, the frequencies won't be as exact as the code language's. But in general, do the frequencies of most words in English match up with the frequencies of their corresponding words in most other languages?

How close are the frequencies of corresponding single words across multiple languages? How close are the frequencies of corresponding pairs of words (for example, the frequency of "John walks" in English and it's equivalent in another language) across languages? Could one theoretically learn a language by comparing the frequencies of words and words in context to that of a known language (mixed with a bit of trial and error in trying to find a definition for each word that fits every context)?

Answer 1

No, it isn't.

Just to take a few examples: The most frequent word in English is the definite article the, but there are languages (e.g., Russian or Chinese) with no definite article at all, and other languages (e.g., German, Italian, French, and Spanish) where the definite article has several different forms according to number, case, and grammatical gender.

Another one: Arabic basically uses just one word for all kinds of saying repeating all over, while in English it is customary and considered good style to vary the word say, ask, answer, reply, even go.

Answer 2

Your question about word-for-word encoding is similar to letter-for-letter encoding. Each European language has its own frequency of letters, so a naive cryptographic encoding (like shifting the alphabet by n letters) is easily cracked by looking at the frequency profile of the encoded text.

---

As a rule, in European languages, the most frequent words are those with a high grammatical role, like be, of, the, I, me, you, in, out, etc. then come those with more semantic and lexical contents. I've not read frequency surveys of sinograms. Chinese is peculiar because the vocabulary is componential and grammar is minimal. So I don't know how different Chinese would appear in that respect.

---

your last question is related to the idea of pivot language. For example, it's obvious translate.google uses English as its pivot language (which causes some distorsion as English does not distinguish P2sg from P2pl). The main problem is that "John walks" is not necessarily translated in real languages as a Noun+Verb sentence. Arabic would say "John walking" with a participle, here in masculine form, and Irish would say "Is John a-walk" with an infinitive. I'm afraid your proposed method would seriously fail ...

Answer 3

Another reason why this would not work is because if you take any fixed number X and learn the X most common words in each language, the % of words you would recognize (by frequency) vary dramatically.

For instance, at 5000 words, the percent you would recognize would be:
96% for French,
93.5% for English,
91.7% for Chinese,
89.3% for Korean,
and.... a mere 81.7% for Japanese.

This is what's making me seriously consider quitting Japanese after having learned 5000 words as it usually takes 95-98% word recognition to be able to effectively learn/understand new words from context.

Even after knowing the most common 10000 words in Japanese, it is said that you would only recognize 90% of the words by frequency, which you could pretty much get to (or surpass) in any other language (even so-called "hard" ones like Chinese and Korean) with only half the vocabulary study. In my case, I realized that getting from my current level to 90% would require roughly the same amount of study as going from 0 in a language with similar grammar (like Korean) to the same 90%. And from there, I'd much sooner reach the 95-98% threshold than I would in Japanese.

Therefore, there is no such one-to-one correlation by frequency. If we assume that Japanese and English express the same core concepts (which I believe you would have to assume for your theory to work out), and we were to assign the 5000 most frequent words of English to the 5000 most frequent words of Japanese, the resulting Japanese encoding would be lower in semantic richness than the source encoding from English.

(Some of these links are in Japanese, but they require minimal scrolling to get to the charts that reveal the information I mentioned. The article that makes the case for 95-98% comprehension is in English and extremely illuminating.)