3

The Zipf mystery: In (presumably all from what I have read) natural languages, the number of times any specific word shows up in the language as a whole, or any particular piece is approximately proportionate to its rank.

For example, the word "The" in English is the #1 most frequently used work, and the word "of" is the second. The word "of" shows up approximately half as many times as the word "the".

These rules apply to every word in English, and almost every other language. Are there any theories that make a good attempt to explain this?

  • 3
    Relevant Wikipedia article. I believe Zipf's law also shows up in other places, not just language. – ewawe Jan 21 '17 at 22:06
  • 1
    Have a look here: linguistics.stackexchange.com/questions/10004/… – fdb Jan 21 '17 at 22:06
  • 1
    Your question of 97 words has 60 different words - top 20 % by rank ( top 12 words ) constitute 70 % of occurrences. The word 'the' occurs 11 times and is highest ranked the second ranked 'word' occurs 6 times. even this small body of text, as we see, was not immune to this law. – ARi Jan 22 '17 at 14:34
  • 1
    BTW, the very top ranks may show significant deviations from Zipf's law, intermediate ranks are much better fits to it. – jknappen - Reinstate Monica Jan 23 '17 at 13:08
  • @jknappen large deviation in linear count and large deviation in terms of percentages aren't the same thing. While top words may differ from the 1/2 by seemingly large amounts, when you look into the percentages, it's not really all that much – Travis Jan 23 '17 at 13:18
2

One research group at our university is particularly interested in the statistical properties of language. One professor, Michael Ramscar, is teaching us some classes this semester on related topics. And basically the idea is that this kind of logarithmic/exponential distribution is considered optimal in an information theory point of view, since it ensures that the entropy of the whole probability distribution would be the lowest, ensuring the most effective communication in general. Some of these ideas can also be found at his blog, for example this article about the distribution of names.

P.S.: Actually he also made the argument that the "Zipfian power law" view is not entirely accurate while an "logarithmic/exponential" distribution might be a better description. He said that the shape of the curve under the Zipfian power law would be largely impacted by the sample size, while the shape of the curve under simple logarithmic/exponential distribution wouldn't, because it would only have one parameter, while the Zipfian power law distribution also gets a scaling factor besides the simple exponential factor. I'm not sure I completely got this point to be honest, but the point about entropy should still stand regardless, since I think Zipfian distribution is basically logarithmic/exponential as well.

  • Why would low entropy be good for information? I thought low entropy = inefficient information transfer in this context. – ewawe Feb 5 '17 at 21:55
  • @sumelic Well, the WP page says "Generally, entropy refers to disorder or uncertainty." Basically, if you need a lot of guesses to get one piece of information right/more bits to convey the same amount of information, then that communication wouldn't be very effective. – xji Feb 5 '17 at 22:21
  • @sumelic See also en.wikipedia.org/wiki/Self-information – xji Feb 5 '17 at 22:22
  • But I thought that is what makes high entropy more efficient for communication. I learn more from surprising messages, where I wouldn't be able to easily guess the identity of some words from the other words, than from messages that I could easily guess the contents of from reading only a small part. – ewawe Feb 5 '17 at 22:23
  • 1
    @sumelic But that might easily confuse people. I remember there was some data about how about more than 100 words constitute more than half of the words used in daily speech. If thousands of words are used with equal probabilities, there will be much more cognitive load and it would be harder for people to reasonably predict/expect what's coming next. Of course people learn new information by surprising messages. But much of the daily conversation isn't about surprise is it. It's about reliably expressing some common set of daily ideas/functionalities. – xji Feb 5 '17 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.