How often a linguistic element occurs relative to other linguistic elements.

learn more… | top users | synonyms

2
votes
1answer
31 views

Probabilities for 2-grams are higher than 1-grams in arpa file produced by kenlm

I'm using the 1 billion word language corpus to build a model with 1 and 2-grams. When using the lmplz program that comes with kenlm, I noticed that the arpa file seems to have higher probabilities ...
2
votes
2answers
61 views

Word and/or syllable frequency data for Lao

I've returned my language focus to Lao now that my travels through Asia have finished and I'm back home. There are not as many or as high quality resources available for Lao as for many other ...
2
votes
1answer
57 views

Comparing frequencies of two corpora

I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
0
votes
0answers
66 views

Mandarin Chinese syllable and tone frequency (not character frequency)

There's plenty of good resources on Chinese character frequency available. But I'm wondering about syllable frequency independent of characters, and also tone frequency both separate and in ...
6
votes
3answers
374 views

Are there natural languages that do not obey Zipf's law?

Is there a natural language which is known not to follow Zipf's law? I'm interested to see if it's really universal. This is what Zipf's law states: Zipf's law states that given some corpus of ...
0
votes
2answers
113 views

Software to form a word frequency list for a Spanish book

I've actually found one called "textstat", but it doesn't recognize different word forms of one word. So are there programs that do detect comía, come, cómelo, comiera as forms of comer?
1
vote
0answers
49 views

Connection between word usage frequency and number of synonyms for the word

At a first glance it seems to be obvious/expectable (at least to some extent) that the more frequently a word is used in language the more synonyms it should have. Can somebody point out whether this ...
4
votes
1answer
252 views

Establishing word frequencies from a corpus

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words. What is a good approach in order to establish the frequency of a given word from ...
1
vote
1answer
149 views

Looking for Thai letter-frequency resource

I'm currently in Thailand and a bit frustrated with the difficult writing system here. Normally after a week in a country I can learn the writing system to a functional degree (except Han characters ...
5
votes
1answer
280 views

Words and phrases more likely in everyday speech

I'm processing a large corpus for a given language. I've noticed that for certain sections of the corpus, I can go many tens of thousands of words without the word "me" being mentioned. These are ...
5
votes
4answers
4k views

How many “words” do I need to learn?

I am interested in learning a second language. To do this I have created a list of the 1000 most common words and phrases for a given language. I've also established sentences which contains each of ...
1
vote
0answers
87 views

What mechanism of language can be used to explain competitive synonyms similar to high-frequency word?

Suppose we have two competitive synonyms A and B. We found out that during the historical changes of the language, A gains dominant position. We found out the fact above via a statistical survey of ...
3
votes
1answer
354 views

Word frequency list for agglutinative languages like Swahili?

I'm an intermediate Swahili speaker looking to substantially increase the size of my vocabulary over the next couple weeks. However, I want to optimize my learning curve by focusing on the first 2000 ...
3
votes
1answer
2k views

Most common english words from newspapers, books and magazines

I'd like to download a frequency list of english words that are most common in newspapers, books and magazines. It should contain at least 50 000 words. Is it possible to download such a list from ...
1
vote
0answers
151 views

Searching for a Spanish word list for statistical analysis [closed]

I need a Spanish word list, as simple as that. The more complete it is, the better, it should contain as many words as to be a statistically relevant sample. It can be in any format: XML, MS Excel, ...
3
votes
1answer
162 views

What is the idea of ngram viewers working case-sensitive? [closed]

On German Language & Usage we recently had a question about the English word schmuck, which is a false friend of the German word Schmuck. Out of curiosity I tried the capitalized form vs the non ...
6
votes
1answer
172 views

Are there tools similar Google ngram, which recognise conjugated or declined word forms?

I just found out, that with respect to the German language, where prefixes of verbs are separated from the verb and posed behind the object, I have no chance to find out, if a verb with a prefix is ...
1
vote
1answer
572 views

What are the most commonly used Chinese syllables?

There are about 1200 or so unique (includes all tones) Mandarin Chinese syllables, according to some source I read a while back based off the Unihan database. For my applications I'm limited to using ...