9

As with all natural laws, Zipf's law is an approximation. If you take a large corpus, and compute the Zipf curve, it will more or less follow a Zipf distribution (with coefficients thrown in to account for slack). This doesn't mean that for every language it follows the exact rule of 'the second most common lexical item is 1/2 as frequent as the most common'...


9

First, just to note, *kweþaną didn't completely die out: English "quoth" is archaic but still recognizable, and Icelandic kveða is still in active use. But you're absolutely right about the general trend that killed off most descendants of *kweþaną. It's now quite rare across the Germanic languages, when it used to be widespread. It's hard to say why ...


8

Zipf’s law, as I understand it, is not really about languages, but about statistics and probability. It is just one of several formulations of the fact that many non-arbitrary sequences of numbers (frequency of words in a given corpus; population size of cites in relation to their rank; annual turnover of ranked companies; etc., etc.) are not evenly ...


6

I am not sure whether your initial assumption is statistically correct, but let us take it as a working hypothesis. French and German (to mention only these) very commonly use "on" and "man" with an active verb where English prefers a passive construction. Thus: "on dit" = "man sagt" = "it is said". Of course you can also say "one says", but this is less ...


5

I've downloaded the Thai Wikipedia dump of the 29/08/2013, converted its 82,200 articles into text files with wp2txt and counted the 195,667,724 characters with a small python script. wp2txt is supposed to remove the metadata, but the statistics can be skewed anyway, and since I don't know any Thai, I have no way to check whether the statistics below make ...


5

Whoops, didn't know I was allowed to answer my own question. Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely proportional to its rank in frequency, so the most common word is used twice as much as the second most common word, three times as much as the third most common, and ...


5

No, it isn't. Just to take a few examples: The most frequent word in English is the definite article the, but there are languages (e.g., Russian or Chinese) with no definite article at all, and other languages (e.g., German, Italian, French, and Spanish) where the definite article has several different forms according to number, case, and grammatical gender....


4

There is definitely a correlation, but it is not absolute. If we imagine a spectrum of values of s (the value of the exponent characterizing the distribution), different language families overlap on the spectrum. The distribution is a function of morphological typology and orthography, and sometimes otherwise very closely related languages differ ...


4

Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word prevalence for 61,800 English words by Marc Brysbaert, Paweł Mandera, Samantha McCormick, and Emmanuel Keuleers in the link given.


4

It is a matter of linguistic pragmatics. A typical statement has a topic, also known as theme or given (what is being talked about) and a comment or rheme (the new information about the topic). In 1843 Weil noticed the tendency of actual language used to reflect a topic - comment word order (what he called 'the march of ideas'); this is especially true when ...


3

The token frequency data you need is vanishingly rare. I think you could add Dutch, and perhaps you can add in a couple more languages like French and Spanish (although you could get pilot estimates for various languages, where a study reports frequency data for one or two non-random texts written in a language). The biggest effect that you would need to ...


3

I found one paper on the Internet which presents a syllabification algorithm for Lao: Syllabification of Lao Script for Line Breaking I don't find it fully describes what it purports to though and it seems to cover syllable structures I'm so far unaware of in Lao.


3

I have found this one, the icu_tokenizer. The icu_tokenizer uses the same Unicode Text Segmentation algorithm as the standard tokenizer, but adds better support for some Asian languages by using a dictionary-based approach to identify words in Thai, Lao, Chinese, Japanese, and Korean, and using custom rules to break Myanmar and Khmer text into syllables. ...


3

Even though you already received a good answer, I'd like to point out Gries' 2008 paper "Dispersions and adjusted frequencies in corpora" which is sort of a must-read for anyone doing corpora linguistics. It's available at http://www.linguistics.ucsb.edu/faculty/stgries/research/2008_STG_Dispersion_IJCL.pdf.


2

Q: A word like the name "Barry" might be very common in one of the corpus files (say a novel) and this will result in a larger than expected frequency for this word if you simply add all of its occurrences in the corpus and divide my 7 million. Many corpora (except very large ones) only include parts of larger texts like novels (such as 2,000 words) to ...


2

Whether you normalise by 100,000 or 1,000,000 words is essentially a question of style and/or common practice in the journal or sub-field you're in. Mathematically, it makes not difference at all whether you say feature x occurs with a frequency of 10 per 100,000 words or 100 per 1,000,000 words. But you should be consistent, so if you provide other ...


2

Regarding corpus size: If you need meaningful results for the less frequent characters, you'll need a larger corpus, such as the ones at http://corpus.byu.edu. Their web interface accepts queries consisting of punctuation. A sample query is .|,|:|;|!|?|'|"|@|#|$|%|^|&|*|(|)|-|~, which for the Contemporary American English, Global Web-based English and ...


2

The paper you linked to describes how the researchers compiled their Academic Vocabulary List (AVL). They first identified a corpus of academic texts, which they extracted from a larger and more general Corpus of Contemporary American English (COCA). They then came up with various measurable criteria by which to select words (or rather lemmas, to be precise) ...


2

This really depends on what you want to do with that information. I'm just in the middle of a project where we're trying to come up with measures of difficulty to give people some automatic help when reading texts. In general, you can say that the more frequent word, the more likely it is that any random person will understand it. But when you say '...


2

What is considered a large enough sample size when being used to guide the creation of pedagogical resources such as foreign language courses and why? How large your corpus should be depends on what exactly you want to use it for, and what alternatives are available. If you want to know the 2,000 most frequent words of a language, a 1 M. word corpus will be ...


2

You can try the Spanish stemming algorithm found at http://snowball.tartarus.org/algorithms/spanish/stemmer.html.


2

There are two "domains", time and frequency. If you are using seconds as the units of measurement, the analysis is in the time domain. If the unit or measurement if Hz, the analysis is in the frequency domain. You are not supposed to analyze one domain as independent of another, indeed "frequency" can only be defined by reference to a unit of time (it recurs ...


2

One research group at our university is particularly interested in the statistical properties of language. One professor, Michael Ramscar, is teaching us some classes this semester on related topics. And basically the idea is that this kind of logarithmic/exponential distribution is considered optimal in an information theory point of view, since it ensures ...


2

If what you really want is Wiktionary's frequency list, you can get that by taking the various Wiktionary pages linked from the overview page and splicing them together. However, be careful which corpus you're using. The idea of a "universal" corpus is generally a myth: comparing the frequencies of these words in TV scripts versus Project Gutenberg versus ...


2

This question hasn't been updated in a while. Google also released a 1 Billion Word Benchmark Corpus: https://github.com/tensorflow/models/tree/master/research/lm_1b. It has a vocabulary size of about 800k, as reported on the repo. The corpus was collected from English language newswire services from all over the world. I've processed the unigram frequency ...


2

Frequent use of the passive in English is not a breach of any "recommended" proportion. Rather, it is a function of register, i.e. it depends on the formality of the situation and the education of the speaker/writer. Some examples: Written English uses more passives than spoken English. In many scientific disciplines, students are told very early ...


2

The Leipzig Wortschatz Portal is a good place to get answers for this kind of questions. The definite article is more than twice as frequent as the indefinite one (158M vs 67M + 10M). Unfortunately, no further breakdown is in there, they just count wordforms, but on very large corpora.


2

Fortunately, the very same source (Whitney's grammar) used in your conjunct consonants link also gives the frequencies of individual phonemes, which I will take to be an acceptable proxy for non-conjunct glyphs. If you are really interested in the frequency of non-conjunct glyphs but not the phonemes, then I can't help, sorry. Here is the relevant page.


1

There is an answer to your question find the pair of words that never appear together, and have the highest individual frequency (the sum of them): Using the Penn Treebank as a text corpus, the top 10 pairs are: (8090, 'the', 'the'), (6364, 'the', 'of'), (6209, 'the', 'to'), (5923, 'the', 'a'), (5923, 'a', 'the'), (5617, 'the', 'in'), (5556, 'the', '...


Only top voted, non community-wiki answers of a minimum length are eligible