5

If someone tried to invent a code where they simply replaced every English word with another word, the code could be cracked (given a large enough sample) by comparing frequencies of English words to the frequencies of the code words. For example, since "the" is the most common English word and if a code-breaker were able to figure out that "dragef" is the most common code word, the code-breaker could know that "dragef" means "the."

The frequencies of each word in the code language will be the same as the frequencies of their corresponding words in English. So, I wonder, is the same true for other real languages? I realize that, because a word from one language rarely has an exact equivalent in another language, the frequencies won't be as exact as the code language's. But in general, do the frequencies of most words in English match up with the frequencies of their corresponding words in most other languages?

How close are the frequencies of corresponding single words across multiple languages? How close are the frequencies of corresponding pairs of words (for example, the frequency of "John walks" in English and it's equivalent in another language) across languages? Could one theoretically learn a language by comparing the frequencies of words and words in context to that of a known language (mixed with a bit of trial and error in trying to find a definition for each word that fits every context)?

  • 1
    That particular scheme wouldn't work with all languages because not all languages work like English; for instance, not all languages have "words" in the same sense as English does. But it's an interesting question when limited to languages structured analytically like English or Chinese or Vietnamese. There would still be problems about determining and word boundaries and meanings, though. – jlawler Feb 7 at 1:22
  • 3
    The distribution is the same (Zipfian), but the words are not. (Not all languages have a separate word for an indefinite article or definite article, some have multiple words.) You may be interested in unsupervised approaches to machine translation, and generally multilingual embeddings. In practice they are not fully unsupervised, they use a few known translations (like URLs) to bootstrap, and they have limitations. – Adam Bittlingmayer Feb 7 at 10:46
4

No, it isn't.

Just to take a few examples: The most frequent word in English is the definite article the, but there are languages (e.g., Russian or Chinese) with no definite article at all, and other languages (e.g., German, Italian, French, and Spanish) where the definite article has several different forms according to number, case, and grammatical gender.

Another one: Arabic basically uses just one word for all kinds of saying repeating all over, while in English it is customary and considered good style to vary the word say, ask, answer, reply, even go.

2

Your question about word-for-word encoding is similar to letter-for-letter encoding. Each European language has its own frequency of letters, so a naive cryptographic encoding (like shifting the alphabet by n letters) is easily cracked by looking at the frequency profile of the encoded text.


As a rule, in European languages, the most frequent words are those with a high grammatical role, like be, of, the, I, me, you, in, out, etc. then come those with more semantic and lexical contents. I've not read frequency surveys of sinograms. Chinese is peculiar because the vocabulary is componential and grammar is minimal. So I don't know how different Chinese would appear in that respect.


your last question is related to the idea of pivot language. For example, it's obvious translate.google uses English as its pivot language (which causes some distorsion as English does not distinguish P2sg from P2pl). The main problem is that "John walks" is not necessarily translated in real languages as a Noun+Verb sentence. Arabic would say "John walking" with a participle, here in masculine form, and Irish would say "Is John a-walk" with an infinitive. I'm afraid your proposed method would seriously fail ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.