Skip to main content

Questions tagged [entropy]

For questions about entropy in language.

Filter by
Sorted by
Tagged with
0 votes
0 answers
91 views

What is the information density of factual knowledge in large bodies of English text?

An ML paper I was reading mentioned an estimate of no more than 0.7 bits per word, in footnote 4: As of February 1, 2024, English Wikipedia contains a total of 4.5 billion words [...] We estimate ...
MWB's user avatar
  • 1,142
1 vote
1 answer
111 views

Do Natural Languages have about 50% Information per Syllable as would be Optimal

I found this article summarizing this paper. The article has an interesting observation which the paper did not touch on: They found that Japanese, which has only 643 syllables, had an information ...
E Tam's user avatar
  • 171
1 vote
0 answers
55 views

What differentiates the usage of MI and MI3 in collocation analysis?

In Brezina (2018), it is stated that to measure collocation strength, one may use the Mutual Information (MI) to test the collocational strength of a given collocation. MI is naturally more sensitive ...
pindakazen's user avatar
-1 votes
1 answer
299 views

Does deep orthography decrease the collision entropy of a written language?

EDIT: For those of you who think this question is off-topic, I asked it on a forum about linguistics because I expected the answer to come from historical linguistics. I expected the answer to be ...
FlatAssembler's user avatar
-3 votes
1 answer
135 views

Perfect Language [closed]

In linguistics, what's the definition of a perfect language? Off the top of my hat, given the primary purpose of language is communication, a perfect language must be able to convey 100% of the ...
Hudjefa's user avatar
  • 107
0 votes
0 answers
104 views

How is collision entropy of German syntax higher than of English one?

As I explained in the draft of my latest paper about linguistics, I have measured the collision entropies of English, German and Croatian syntax. The entropy of consonants in a long English text was ...
FlatAssembler's user avatar
2 votes
0 answers
33 views

Database of Words and their Semantic Entropies?

I'm reading up on semantic entropies and it seems as though there aren't any actual databases of words and their associated semantic entropies for any given methodology. For instance, this study has a ...
Josh Tint's user avatar
3 votes
2 answers
161 views

How to quantify the semantic depth of a sentence?

I'm looking for a robust way to roughly quantify the amount of information conveyed in a sentence, specifically in English. For instance "He went to the place" conveys less information than &...
Josh Tint's user avatar
3 votes
0 answers
75 views

Computational linguist's estimate of minimum hours of speech needed to train a neural net to recognize speech

From a computational linguist's point of view, is there a lower limit on the number of hours of speech needed to train a neural net to translate speech to text? An estimate from CMU is 3000-5000 hours ...
Lars Ericson's user avatar
7 votes
3 answers
528 views

Speech rate measured in bit/sec

Are there references giving the speech rate measured in bit/sec? I easily found references on speech rate giving measurement in phones/sec, syllables/sec or words/sec. Are there references giving the ...
Sir Cornflakes's user avatar
5 votes
1 answer
672 views

Shannon's entropy as a measure of vocabulary richness

The Entropy formula for lexical richness is The probability p-ith is calculated by dividing V-ith by N, where N is the total number of tokens in the text and V is the number of types. However, I can'...
Peter's user avatar
  • 111
12 votes
2 answers
3k views

What is the entropy per word of random yet grammatical text?

It has been suggested that to construct an uncrackable yet memorable passphrase with 256-bits of entropy, the passphrase should be manifested as a poem. The answerer made an estimation of the entropy ...
user avatar