Can someone explain to me the Zipf–Mandelbrot law?

Question

I really can't understand. It's about linguistics and I can't understand anything because there are mathematical formulas in it that I can't understand at all. Can anyone explain this with hypothetical examples? Thanks in advance!

Answer 1

As this is the Linguistics site, I'm guessing it is Zipf's Law you are interested in? If you analyze a corpus of text, and count the number of times each word occurs, you get their frequency. You then line then up on the x-axis of a graph, in order of rank (i.e. the most common word comes at x=1, the second most common word at x=2, and the least common word all the way over on the right at x=N), and their frequency on the y-axis.

Zipf's Law just says that the frequency is inversely proportional to their rank. So, if the most common word occurred 1000 times, you would expect the second-most common word to occur 500 times, the 3rd to occur 333 times, the 4th-ranked word to occur 250 times. And over on the right you would expect all the words from rank 667 (*1) to rank 2000 (*2) to occur exactly once.

*1: 1000/666 is 1.5015 so would round up to 2, and you expect it to occur twice.

*2: 1000/2001 is 0.4998 so would round down to 0; that means you expect only 2000 words to be used in your corpus.

Obviously a real corpus won't exactly match, but you could reasonably expect it to have roughly the same shaped curve. If not, there may be something unusual or artificial about your corpus. E.g. A glossary or index would not follow the curve very well.

Zipf-Mandelbrot is a generalization of Zipf's Law, adding some parameters so a single law can be used to describe a range of probability distributions.