4

The Zipf mystery: In (presumably all from what I have read) natural languages, the number of times any specific word shows up in the language as a whole, or any particular piece is approximately proportionate to its rank.

For example, the word "The" in English is the #1 most frequently used work, and the word "of" is the second. The word "of" shows up approximately half as many times as the word "the".

These rules apply to every word in English, and almost every other language. Are there any theories that make a good attempt to explain this?

Improve this question
5
  • 3
    Relevant Wikipedia article. I believe Zipf's law also shows up in other places, not just language.
    – brass tacks
    Commented Jan 21, 2017 at 22:06
  • 1
    Have a look here: linguistics.stackexchange.com/questions/10004/…
    – fdb
    Commented Jan 21, 2017 at 22:06
  • 1
    Your question of 97 words has 60 different words - top 20 % by rank ( top 12 words ) constitute 70 % of occurrences. The word 'the' occurs 11 times and is highest ranked the second ranked 'word' occurs 6 times. even this small body of text, as we see, was not immune to this law.
    – ARi
    Commented Jan 22, 2017 at 14:34
  • 1
    BTW, the very top ranks may show significant deviations from Zipf's law, intermediate ranks are much better fits to it.
    – Sir Cornflakes
    Commented Jan 23, 2017 at 13:08
  • @jknappen large deviation in linear count and large deviation in terms of percentages aren't the same thing. While top words may differ from the 1/2 by seemingly large amounts, when you look into the percentages, it's not really all that much
    – Travis
    Commented Jan 23, 2017 at 13:18

1 Answer 1

Reset to default
5

One research group at our university is particularly interested in the statistical properties of language. One professor, Michael Ramscar, is teaching us some classes this semester on related topics. And basically the idea is that this kind of logarithmic/exponential distribution is considered optimal in an information theory point of view, since it ensures that the entropy of the whole probability distribution would be the lowest, ensuring the most effective communication in general. Some of these ideas can also be found at his blog, for example this article about the distribution of names.

P.S.: Actually he also made the argument that the "Zipfian power law" view is not entirely accurate while an "logarithmic/exponential" distribution might be a better description. He said that the shape of the curve under the Zipfian power law would be largely impacted by the sample size, while the shape of the curve under simple logarithmic/exponential distribution wouldn't, because it would only have one parameter, while the Zipfian power law distribution also gets a scaling factor besides the simple exponential factor. I'm not sure I completely got this point to be honest, but the point about entropy should still stand regardless, since I think Zipfian distribution is basically logarithmic/exponential as well.

Improve this answer
8
  • 1
    Why would low entropy be good for information? I thought low entropy = inefficient information transfer in this context.
    – brass tacks
    Commented Feb 5, 2017 at 21:55
  • @sumelic Well, the WP page says "Generally, entropy refers to disorder or uncertainty." Basically, if you need a lot of guesses to get one piece of information right/more bits to convey the same amount of information, then that communication wouldn't be very effective.
    – xji
    Commented Feb 5, 2017 at 22:21
  • @sumelic See also en.wikipedia.org/wiki/Self-information
    – xji
    Commented Feb 5, 2017 at 22:22
  • But I thought that is what makes high entropy more efficient for communication. I learn more from surprising messages, where I wouldn't be able to easily guess the identity of some words from the other words, than from messages that I could easily guess the contents of from reading only a small part.
    – brass tacks
    Commented Feb 5, 2017 at 22:23
  • 2
    @sumelic But that might easily confuse people. I remember there was some data about how about more than 100 words constitute more than half of the words used in daily speech. If thousands of words are used with equal probabilities, there will be much more cognitive load and it would be harder for people to reasonably predict/expect what's coming next. Of course people learn new information by surprising messages. But much of the daily conversation isn't about surprise is it. It's about reliably expressing some common set of daily ideas/functionalities.
    – xji
    Commented Feb 5, 2017 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.