14

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal.

This is what Zipf's law states:

Zipf's law states that given some corpus of natural language utterances, the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc.

Edit - To clarify a bit - imagine that you could snoop on every conversation held by native speakers everywhere, in every context, over the course of one year, and record every word, and then count the words occurring, and calculate their frequencies, would they deviate from Zipf's law? Now, we can't do that in reality, but for languages with a substantial written corpus, we can have a broad enough sample.

Improve this question
16
  • 5
    Surely this is missing the point of Zipf's law? It's not meant to be a linguistic universal in the same way that, say, the binding conditions might be. It doesn't apply to languages, but to corpora, and no matter how big your corpus, you can never equate that corpus with the language it's written in. Asking, e.g. "Does Portuguese obey Zipf's law?" is meaningless.
    – P Elliott
    Commented Jan 12, 2014 at 15:44
  • Does that mean word frequencies differ between the spoken and written language?
    – sashoalm
    Commented Jan 12, 2014 at 16:33
  • 1
    Sasholm, I don't agree with P Elliot about his intent of corpus vs language. A sample vs the entirety/rule that produces them is a well-known statistical concept. The Zipf curve for Finnegan's Wake is well known to be noticeably skew from the Zipf curve (as well as the opposite direction from, say, low complexity text like the Twilight series). All human languages follow Zipf's law more or less, none exactly.
    – Mitch
    Commented Jan 13, 2014 at 13:41
  • 2
    @PElliott Samples (which are always a finite set) can come from a discrete or continuous space; the continuous space is necessarily infinite but the discrete space can also. Anyway, with respect to language, how do you come up with a -grammatical- rule like 'articles come before a noun', which is about an infinite number of sentences, from only a finite sampling of heard utterances?
    – Mitch
    Commented Jan 14, 2014 at 14:26
  • 1
    @PElliott sure, but that's not the point. Both the linguist (explicitly) and the language learner (implicitly) are creating universal rules from a finite set of data. So the distinction between language and corpora doesn't help. A Zipf curve is calculated from a corpus, but Zipf's law is about the language.
    – Mitch
    Commented Jan 14, 2014 at 19:36

4 Answers 4

Reset to default
10

As with all natural laws, Zipf's law is an approximation. If you take a large corpus, and compute the Zipf curve, it will more or less follow a Zipf distribution (with coefficients thrown in to account for slack).

This doesn't mean that for every language it follows the exact rule of 'the second most common lexical item is 1/2 as frequent as the most common'. It's just a lax observation. One can do a regression analysis to discover exactly the coefficients for a particular language.

Even within a language there are divergences. It isn't hard to find works whose Zipf curves diverge from the general language's. Joyce's Finnegan's Wake uses so many rare and made-up words that its tail is thick and long. But children's literature attempts to be easily understood and so has few rare words and drops off much more sharply.

Zipf's law doesn't just approximate word frequencies but also letter frequencies, city sizes, income ranks, and many other rank vs. frequency graphs. It is taken into account in decrypting substitution ciphers, and also in creating codes (artificial languages) that do not follow Zipf's law.

Improve this answer
5
  • 2
    True, except that the harmonic pattern of city sizes was not discovered by Zipf but much earlier by Auerbach. Credit where credit is due.
    – fdb
    Commented Jan 14, 2014 at 23:52
  • I know very little statistics. Is there a standard measure of how closely does a given corpus follow an ideal Zipf curve (e.g. a number that would be high for a typical text, and low for Finnegan's Wake)?
    – melissa_boiko
    Commented Sep 2, 2016 at 14:07
  • This answer is recapping what the Zipf law is, and saying that not Zipf law are the same. Of course they are not, as there's a free parameter in the Zipf law. But it does not really answer the question, which I find very interesting.
    – famargar
    Commented Mar 2, 2017 at 9:31
  • @famargar Good point that I did not exactly outright answer the question. The question is essentially an empirical or statistical one - I don't know of a test of all current languages as to how close they fit to the Zipf law. So I don't know the answer. I just highly suspect that all natural languages are roughly Zipf-like because a human language would have to be pretty strange to not follow it.
    – Mitch
    Commented Mar 2, 2017 at 14:53
  • 1
    Also the distribution of income was already captured in the Lorentz curve, when Zipf was three years old. (@fdb)
    – Keelan
    Commented Sep 7, 2017 at 5:51
8

Zipf’s law, as I understand it, is not really about languages, but about statistics and probability. It is just one of several formulations of the fact that many non-arbitrary sequences of numbers (frequency of words in a given corpus; population size of cites in relation to their rank; annual turnover of ranked companies; etc., etc.) are not evenly distributed along a decimal scale, but are more or less evenly distributed along a logarithmic scale. As such, it ought to work with all language corpora, including texts that avoid the use of certain letters.

Improve this answer
1
  • 1
    It works for finate probabalistic spaces only (texts and corpora), not for languages.
    – Manjusri
    Commented Jan 13, 2014 at 9:28
1

Spanish doesn't follow it. Not even remotely.

https://en.wikipedia.org/wiki/Most_common_words_in_Spanish

Improve this answer
3
  • 1
    Hmm interesting. See however this graph of frequency vs rank for 30 Wikipedia languages where Spanish follows the trend as much as any other.
    – curiousdannii ♦
    Commented Jan 30, 2018 at 3:51
  • 1
    But if you add up all the articles it does come closer. Maybe Zipf's law should be applied to uninflected roots.
    – curiousdannii ♦
    Commented Jan 30, 2018 at 3:54
  • If you use roots instead of all words, the curve looks much the same. If you separate the curves by Part-Of-Speech, you will notice how the left side flattens (due to elided words), and the slopes vary between open and closed classes.
    – amI
    Commented Feb 2, 2018 at 22:58
-1

A sample of any text written constraitively and/or according to grammars of avoidance speech styles (including honorific speech styles for languages with honorific lexemes but without neutrally polite speech styles) would contradict the Zipf's law.

This can be best shown by the phonetical and lipogrammatical example of A Void by Perec. This novel has been written without E, the most frequently used French letter, and hence has none of the frequently used words in French vocabulary (e.g. it contains no de and des, which could have the highest freequencies in French texts elsewhere).

In short, the conversation would present a large diversity in lexical occurrence due to the very existance of ideolects.

Improve this answer
11
  • 1
    I'm interested about the second part of you answer. Does that mean that French deviate from Zipf's law? That is, does the entire known written corpus of French also deviate statistically in the word-frequencies?
    – sashoalm
    Commented Jan 11, 2014 at 16:17
  • 1
    I've edited my question to clarify what I mean. In particular, the novel "A Void", as I understand it, deviates from Zipf's law, but it also deviates from the language-wide statistical distribution for French.
    – sashoalm
    Commented Jan 11, 2014 at 21:21
  • 3
    Maybe someone should actually do a word count on Perec’s “La disparition”. I would not be surprised if it did conform with Zipf’s law.
    – fdb
    Commented Jan 11, 2014 at 22:59
  • 2
    The question was about particular languages in their totality, not about specific texts in those languages. I can write a three volume text consisting of a single recurring word, kind of "Buffalo buffalo buffalo ...", but it won't have anything to do with the general tendencies of English.
    – Yellow Sky
    Commented Jan 11, 2014 at 23:35
  • 2
    @YellowSky it doesn't make sense to ask about Zipf's law wrt to a particular language in its totality, since the totality of a language is an infinite set of sentence, and an infinite set of sentences by definition will fail to meet Zipf's law.
    – P Elliott
    Commented Jan 13, 2014 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.