# Do Natural Languages have about 50% Information per Syllable as would be Optimal

I found this article summarizing this paper. The article has an interesting observation which the paper did not touch on:

They found that Japanese, which has only 643 syllables, had an information density of about 5 bits per syllable, whereas English, with its 6949 syllables, had a density of just over 7 bits per syllable. Vietnamese, with its complex system of six tones (each of which can further differentiate a syllable), topped the charts at 8 bits per syllable.

If you have N options, then you can at most transmit I information per decision, where I = log2(N) [source]. If you consider Vietnamese having 17,974 syllables, then Japanese, English, and Vietnamese could optimally transmit 9 bits, 12 bits, and 14 bits per syllable, respectively. That is about double what these researchers measured each language actually does transmit per syllable.

I wondered if this trend could be universal. However, when I looked for answers I kept winding up back at the paper linked above, which is actually about a different topic. Does anyone have any insight on this?

EDIT: I had an idea and dug though the paper's supplementary material and they do happen to list how many syllables each of their studied languages have. I combined that with their other data in a spreadsheet and got this:

ShE is the Shannon entropy and ID is the conditional entropy. It appears that the ratio for those measurements to the maximum possible does stay around 0.75 and 0.5, respectively. Does anyone know if there is a formal study of this, or a theoretical explanation for why those ratios have those values?

• Note that the information density in that paper is the conditional entropy, H(X_n|X_n-1), not the Shannon entropy, H(x).
– Draconis
Feb 8 at 2:06
• I know, but it actually does not matter. Conditional entropy is equal to the Shannon entropy under optimal condition (i.e. an uniform probability distribution), so I just wrote the formula for Shannon entropy. Feb 8 at 2:28
• @ETam higher conditional information density is far from optimal, because loss of any segments is harder to recover from. By adding redundancy you increase the likelihood the message can be recovered over a lossy medium, but reduce the information density Feb 8 at 9:22
• If you consider Vietnamese as having 17,794 syllables, then you’d also have to consider English as having far more than 6,949 syllables. 17,794 is the number of that are phonotactically possible in Vietnamese, but as the article says, less than half of those are found in actual words in the language. A comparative count for English would be much trickier because there are many cases where we don’t fully know whether an absent syllable is accidental or due to a blocking rule (e.g., /blɪD/ doesn’t exist, where D = coronal stop), but could potentially yield a number greater than 100,000. Feb 9 at 23:50