Are there any good theories explaining the language aspect of the Zipf Mystery?

The Zipf mystery: In (presumably all from what I have read) natural languages, the number of times any specific word shows up in the language as a whole, or any particular piece is approximately proportionate to its rank.

For example, the word "The" in English is the #1 most frequently used work, and the word "of" is the second. The word "of" shows up approximately half as many times as the word "the".

These rules apply to every word in English, and almost every other language. Are there any theories that make a good attempt to explain this?

• Relevant Wikipedia article. I believe Zipf's law also shows up in other places, not just language. Jan 21 '17 at 22:06
• Have a look here: linguistics.stackexchange.com/questions/10004/…
– fdb
Jan 21 '17 at 22:06
• Your question of 97 words has 60 different words - top 20 % by rank ( top 12 words ) constitute 70 % of occurrences. The word 'the' occurs 11 times and is highest ranked the second ranked 'word' occurs 6 times. even this small body of text, as we see, was not immune to this law.
– ARi
Jan 22 '17 at 14:34
• BTW, the very top ranks may show significant deviations from Zipf's law, intermediate ranks are much better fits to it. Jan 23 '17 at 13:08
• @jknappen large deviation in linear count and large deviation in terms of percentages aren't the same thing. While top words may differ from the 1/2 by seemingly large amounts, when you look into the percentages, it's not really all that much Jan 23 '17 at 13:18

One research group at our university is particularly interested in the statistical properties of language. One professor, Michael Ramscar, is teaching us some classes this semester on related topics. And basically the idea is that this kind of logarithmic/exponential distribution is considered optimal in an information theory point of view, since it ensures that the entropy of the whole probability distribution would be the lowest, ensuring the most effective communication in general. Some of these ideas can also be found at his blog, for example this article about the distribution of names.

P.S.: Actually he also made the argument that the "Zipfian power law" view is not entirely accurate while an "logarithmic/exponential" distribution might be a better description. He said that the shape of the curve under the Zipfian power law would be largely impacted by the sample size, while the shape of the curve under simple logarithmic/exponential distribution wouldn't, because it would only have one parameter, while the Zipfian power law distribution also gets a scaling factor besides the simple exponential factor. I'm not sure I completely got this point to be honest, but the point about entropy should still stand regardless, since I think Zipfian distribution is basically logarithmic/exponential as well.

• Why would low entropy be good for information? I thought low entropy = inefficient information transfer in this context. Feb 5 '17 at 21:55
• @sumelic Well, the WP page says "Generally, entropy refers to disorder or uncertainty." Basically, if you need a lot of guesses to get one piece of information right/more bits to convey the same amount of information, then that communication wouldn't be very effective.
– xji
Feb 5 '17 at 22:21