9

As with all natural laws, Zipf's law is an approximation. If you take a large corpus, and compute the Zipf curve, it will more or less follow a Zipf distribution (with coefficients thrown in to account for slack). This doesn't mean that for every language it follows the exact rule of 'the second most common lexical item is 1/2 as frequent as the most common'...


9

First, just to note, *kweþaną didn't completely die out: English "quoth" is archaic but still recognizable, and Icelandic kveða is still in active use. But you're absolutely right about the general trend that killed off most descendants of *kweþaną. It's now quite rare across the Germanic languages, when it used to be widespread. It's hard to say why ...


8

Zipf’s law, as I understand it, is not really about languages, but about statistics and probability. It is just one of several formulations of the fact that many non-arbitrary sequences of numbers (frequency of words in a given corpus; population size of cites in relation to their rank; annual turnover of ranked companies; etc., etc.) are not evenly ...


5

I've downloaded the Thai Wikipedia dump of the 29/08/2013, converted its 82,200 articles into text files with wp2txt and counted the 195,667,724 characters with a small python script. wp2txt is supposed to remove the metadata, but the statistics can be skewed anyway, and since I don't know any Thai, I have no way to check whether the statistics below make ...


5

No, it isn't. Just to take a few examples: The most frequent word in English is the definite article the, but there are languages (e.g., Russian or Chinese) with no definite article at all, and other languages (e.g., German, Italian, French, and Spanish) where the definite article has several different forms according to number, case, and grammatical gender....


4

Whoops, didn't know I was allowed to answer my own question. Zipf wrote about exactly this! I knew that he'd formulated Zipf's Law (the relative frequency of a word in a language is inversely proportional to its rank in frequency, so the most common word is used twice as much as the second most common word, three times as much as the third most common, and ...


4

There is definitely a correlation, but it is not absolute. If we imagine a spectrum of values of s (the value of the exponent characterizing the distribution), different language families overlap on the spectrum. The distribution is a function of morphological typology and orthography, and sometimes otherwise very closely related languages differ ...


4

Word frequency is only a proxy for word knowledge. For the English language, there are data available on word prevalence, i.e., on how many people know a certain word. You can find Measures of word prevalence for 61,800 English words by Marc Brysbaert, Paweł Mandera, Samantha McCormick, and Emmanuel Keuleers in the link given.


3

The token frequency data you need is vanishingly rare. I think you could add Dutch, and perhaps you can add in a couple more languages like French and Spanish (although you could get pilot estimates for various languages, where a study reports frequency data for one or two non-random texts written in a language). The biggest effect that you would need to ...


3

I found one paper on the Internet which presents a syllabification algorithm for Lao: Syllabification of Lao Script for Line Breaking I don't find it fully describes what it purports to though and it seems to cover syllable structures I'm so far unaware of in Lao.


3

I have found this one, the icu_tokenizer. The icu_tokenizer uses the same Unicode Text Segmentation algorithm as the standard tokenizer, but adds better support for some Asian languages by using a dictionary-based approach to identify words in Thai, Lao, Chinese, Japanese, and Korean, and using custom rules to break Myanmar and Khmer text into syllables. ...


3

Even though you already received a good answer, I'd like to point out Gries' 2008 paper "Dispersions and adjusted frequencies in corpora" which is sort of a must-read for anyone doing corpora linguistics. It's available at http://www.linguistics.ucsb.edu/faculty/stgries/research/2008_STG_Dispersion_IJCL.pdf.


2

Q: A word like the name "Barry" might be very common in one of the corpus files (say a novel) and this will result in a larger than expected frequency for this word if you simply add all of its occurrences in the corpus and divide my 7 million. Many corpora (except very large ones) only include parts of larger texts like novels (such as 2,000 words) to ...


2

Whether you normalise by 100,000 or 1,000,000 words is essentially a question of style and/or common practice in the journal or sub-field you're in. Mathematically, it makes not difference at all whether you say feature x occurs with a frequency of 10 per 100,000 words or 100 per 1,000,000 words. But you should be consistent, so if you provide other ...


2

Regarding corpus size: If you need meaningful results for the less frequent characters, you'll need a larger corpus, such as the ones at http://corpus.byu.edu. Their web interface accepts queries consisting of punctuation. A sample query is .|,|:|;|!|?|'|"|@|#|$|%|^|&|*|(|)|-|~, which for the Contemporary American English, Global Web-based English and ...


2

The paper you linked to describes how the researchers compiled their Academic Vocabulary List (AVL). They first identified a corpus of academic texts, which they extracted from a larger and more general Corpus of Contemporary American English (COCA). They then came up with various measurable criteria by which to select words (or rather lemmas, to be precise) ...


2

This really depends on what you want to do with that information. I'm just in the middle of a project where we're trying to come up with measures of difficulty to give people some automatic help when reading texts. In general, you can say that the more frequent word, the more likely it is that any random person will understand it. But when you say '...


2

What is considered a large enough sample size when being used to guide the creation of pedagogical resources such as foreign language courses and why? How large your corpus should be depends on what exactly you want to use it for, and what alternatives are available. If you want to know the 2,000 most frequent words of a language, a 1 M. word corpus will be ...


2

You can try the Spanish stemming algorithm found at http://snowball.tartarus.org/algorithms/spanish/stemmer.html.


2

There are two "domains", time and frequency. If you are using seconds as the units of measurement, the analysis is in the time domain. If the unit or measurement if Hz, the analysis is in the frequency domain. You are not supposed to analyze one domain as independent of another, indeed "frequency" can only be defined by reference to a unit of time (it recurs ...


2

One research group at our university is particularly interested in the statistical properties of language. One professor, Michael Ramscar, is teaching us some classes this semester on related topics. And basically the idea is that this kind of logarithmic/exponential distribution is considered optimal in an information theory point of view, since it ensures ...


2

If what you really want is Wiktionary's frequency list, you can get that by taking the various Wiktionary pages linked from the overview page and splicing them together. However, be careful which corpus you're using. The idea of a "universal" corpus is generally a myth: comparing the frequencies of these words in TV scripts versus Project Gutenberg versus ...


2

This question hasn't been updated in a while. Google also released a 1 Billion Word Benchmark Corpus: https://github.com/tensorflow/models/tree/master/research/lm_1b. It has a vocabulary size of about 800k, as reported on the repo. The corpus was collected from English language newswire services from all over the world. I've processed the unigram frequency ...


2

Your question about word-for-word encoding is similar to letter-for-letter encoding. Each European language has its own frequency of letters, so a naive cryptographic encoding (like shifting the alphabet by n letters) is easily cracked by looking at the frequency profile of the encoded text. As a rule, in European languages, the most frequent words are ...


2

The Leipzig Wortschatz Portal is a good place to get answers for this kind of questions. The definite article is more than twice as frequent as the indefinite one (158M vs 67M + 10M). Unfortunately, no further breakdown is in there, they just count wordforms, but on very large corpora.


2

Fortunately, the very same source (Whitney's grammar) used in your conjunct consonants link also gives the frequencies of individual phonemes, which I will take to be an acceptable proxy for non-conjunct glyphs. If you are really interested in the frequency of non-conjunct glyphs but not the phonemes, then I can't help, sorry. Here is the relevant page.


1

There is an answer to your question find the pair of words that never appear together, and have the highest individual frequency (the sum of them): Using the Penn Treebank as a text corpus, the top 10 pairs are: (8090, 'the', 'the'), (6364, 'the', 'of'), (6209, 'the', 'to'), (5923, 'the', 'a'), (5923, 'a', 'the'), (5617, 'the', 'in'), (5556, 'the', '...


1

[EDIT: It seems I may have misunderstood your question. If monosyllables are, indeed, what you are looking for, please, ignore anything below that pertains to word-length. The rest should make some sense regardless.] I don't think there's a simple answer, as there are quite a lot of factors to consider (both language-specific and [quasi-]universal). I can ...


1

Here is one that looks well-researched (starting on p. 214 with the appendix): The high frequency collocations of spoken and written English This one is only for spoken English: Beyond single words - The most frequent collocations in spoken English


1

A lot of things have to be clarified in your question. First, you need to say what the theoretical population of fundamental frequencies is – do you mean "in all instances of speech" (excluding musical and other non-speech uses of phonation)? You also need to specify age range, since infant speech has kind of high F0. You need to specify language: pitch ...


Only top voted, non community-wiki answers of a minimum length are eligible