3

"Zipf's law" is just a pretentious way of saying that many types of data, in various sciences, fit certain kinds of power law distribution. E.g. in linguistics, for a corpus of English word frequency is inversely proportional to the word's rank in frequency (the most common word occurs twice as often as the second most common word, three times as often as the third most common word, etc.). I'm assuming that letter frequency, bigram frequency, trigram frequency, phoneme frequency, etc. are also power-law distributed.

What I'm asking isn't especially to do with the distributions being power laws. I want to know: are the rankings of word frequencies, and other frequencies, more similar (in whatever precise sense of similarity that linguists and statisticians use), between languages which are more closely related?(E.g. are more closely related languages, more likely to have the same most-common phoneme?)

4

There is definitely a correlation, but it is not absolute. If we imagine a spectrum of values of s (the value of the exponent characterizing the distribution), different language families overlap on the spectrum.

The distribution is a function of morphological typology and orthography, and sometimes otherwise very closely related languages differ fundamentally with regard to these.

Morphological Typology

For example English lost most morphology and gender compared to earlier Germanic languages and generally IE languages. A famous example for Zipf's law is English the vs Spanish el, la, los, los, del.... As that cycle completes some grammaticalised words may become affixes like English 'll, or syntax may change as was the case with French shifting to non-pro-drop.

Orthography

Hindustani has two scripts. Chinese has many homographs when written in Chinese, but many homophones when written in Pinyin. In some Hebrew corpora, vowels are indicated, in others only when necessary, resulting in more homographs. German writes noun compounds together, but modern English usually does not. The space character was an innovation that is still not universal, and the definition of a word or token is a matter of debate.

So English and Chinese may have distributions more similar to each other than English and Celtic, or Chinese and Old Chinese, even though English and Chinese are completely unrelated, whereas the other pairs are in the same family.

That said, some of these typological changes happen because of creolisation or substrate, so there are open questions about whether these languages are not in a different family than their nominal ancestors. But even if we assume English is a fusion language, it is a fusion of IE languages.

This is all assuming that the corpora used are actually comparable. English and Scots are very similar languages, but if we use each language's Wikipedia as a corpus to generate the distribution, then there are other variables.

| improve this answer | |
  • 2
    The answer would profit from some data illustrating the claims made here. – jk - Reinstate Monica Feb 20 '18 at 12:58
  • @jknappen Agree. The raw data that I have at hand for 100+ languages is unfortunately Wikipedia. In all datasets I have seen, and in the papers I have read, Hungarian and Turkish have much larger vocabularies ie unique word counts than IE European languages. They are not directly comparable datasets, but it happens every time if we adjust for total corpus size, and it is not surprising. In the two papers I found, the authors even suggest that the shape of the curve is subtly different. Is it the loose correlation claim that you doubt? Or the variance within the families? – Adam Bittlingmayer Feb 21 '18 at 12:04
  • The tokenisation decisions are also subjective, I would probably want to see languages tokenised under multiple schemes to get two bounds. – Adam Bittlingmayer Feb 21 '18 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.