5

The Entropy formula for lexical richness is

enter image description here

The probability p-ith is calculated by dividing V-ith by N, where N is the total number of tokens in the text and V is the number of types. However, I can't seem to grasp the concept of V-ith? Any clarification would be greatly appreciated.

Source: Dale, Moisl, and Somers (p.551). "Handbook of Natural Language Processing" (2000).

Improve this question
5
  • Edition 1 or 2 of the book?
    – Franck Dernoncourt
    Commented Sep 1, 2015 at 21:22
  • @FranckDernoncourt books.google.at/… This one.
    – Peter
    Commented Sep 2, 2015 at 8:43
  • This exact question has been asked at stackoverflow.com/questions/32350202/…. It is still waiting for an answer, too, but the question provides more information.
    – Adam_G
    Commented Sep 2, 2015 at 12:46
  • @Adam_G I'm also the one who asked the other question on stakoverflow, but it doesn't seem like it's getting an answer anytime soon.
    – Peter
    Commented Sep 2, 2015 at 14:12
  • V(i) seems to be the number of tokens of type (i). When v(i) is devided by N (total number of tokens) it yields an estimation for the probability of the type (i) [p(i) = V(i) / N].
    – Claude
    Commented Sep 4, 2015 at 10:21

1 Answer 1

Reset to default
1

The entropy formula as quoted has some ideosyncrasies making it different from standard Shannon entropy

  • There is a (not really relevant) factor of 100 (probably to produce more beautiful numbers)
  • The term log N in the denominator is absent in Shannon entropy

The term p(i) cannot mean anything but the frequency of the i-th type, obtained by dividing the number of occurences of it by the total number of tokens. The frequency equals the probability in a bag-of-words (or more general, a bag-of-types) model.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.