I found this article summarizing this paper. The article has an interesting observation which the paper did not touch on:

They found that Japanese, which has only 643 syllables, had an information density of about 5 bits per syllable, whereas English, with its 6949 syllables, had a density of just over 7 bits per syllable. Vietnamese, with its complex system of six tones (each of which can further differentiate a syllable), topped the charts at 8 bits per syllable.

If you have N options, then you can at most transmit I information per decision, where I = log2(N) [source]. If you consider Vietnamese having 17,974 syllables, then Japanese, English, and Vietnamese could optimally transmit 9 bits, 12 bits, and 14 bits per syllable, respectively. That is about double what these researchers measured each language actually does transmit per syllable.

I wondered if this trend could be universal. However, when I looked for answers I kept winding up back at the paper linked above, which is actually about a different topic. Does anyone have any insight on this?

EDIT: I had an idea and dug though the paper's supplementary material and they do happen to list how many syllables each of their studied languages have. I combined that with their other data in a spreadsheet and got this:

Screenshot of data

ShE is the Shannon entropy and ID is the conditional entropy. It appears that the ratio for those measurements to the maximum possible does stay around 0.75 and 0.5, respectively. Does anyone know if there is a formal study of this, or a theoretical explanation for why those ratios have those values?

  • 1
    Note that the information density in that paper is the conditional entropy, H(X_n|X_n-1), not the Shannon entropy, H(x).
    – Draconis
    Feb 8 at 2:06
  • I know, but it actually does not matter. Conditional entropy is equal to the Shannon entropy under optimal condition (i.e. an uniform probability distribution), so I just wrote the formula for Shannon entropy.
    – E Tam
    Feb 8 at 2:28
  • @ETam higher conditional information density is far from optimal, because loss of any segments is harder to recover from. By adding redundancy you increase the likelihood the message can be recovered over a lossy medium, but reduce the information density
    – Tristan
    Feb 8 at 9:22
  • 1
    If you consider Vietnamese as having 17,794 syllables, then you’d also have to consider English as having far more than 6,949 syllables. 17,794 is the number of that are phonotactically possible in Vietnamese, but as the article says, less than half of those are found in actual words in the language. A comparative count for English would be much trickier because there are many cases where we don’t fully know whether an absent syllable is accidental or due to a blocking rule (e.g., /blɪD/ doesn’t exist, where D = coronal stop), but could potentially yield a number greater than 100,000. Feb 9 at 23:50

1 Answer 1


To get a better intuition: To get at the optimal entropy value, the syllable probability distribution must equal the uniform distribution. Natural languages don't behave like that, instead the distributions of almost anything in natural languages are power-law distributions of which the (in)famous Zipf distribution is an example. So it is clear, that the syllable entropy for natural languages is lower than the theoretical optimum. It should not be too much lower (languages are efficient, after all) and thus the range in the sample looks as expected.

I don't see any a priori reason that the syllable entropy measured as a fraction of the theoretical entropy should assume a certain value, and that this value should be the same for all languages. I also see some problem in the formulation of the problem: Should N, the total number of syllables, be based on observed syllables in a given language (corpus, dictionary, whatever) or based on the "allowed" syllables. Think of English and the word wug that is a well-formed syllable under English phonotactics, but that does not occur in the English language.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.