The Entropy formula for lexical richness is

enter image description here

The probability p-ith is calculated by dividing V-ith by N, where N is the total number of tokens in the text and V is the number of types. However, I can't seem to grasp the concept of V-ith? Any clarification would be greatly appreciated.

Source: Dale, Moisl, and Somers (p.551). "Handbook of Natural Language Processing" (2000).

  • Edition 1 or 2 of the book? – Franck Dernoncourt Sep 1 '15 at 21:22
  • @FranckDernoncourt books.google.at/… This one. – Peter Sep 2 '15 at 8:43
  • This exact question has been asked at stackoverflow.com/questions/32350202/…. It is still waiting for an answer, too, but the question provides more information. – Adam_G Sep 2 '15 at 12:46
  • @Adam_G I'm also the one who asked the other question on stakoverflow, but it doesn't seem like it's getting an answer anytime soon. – Peter Sep 2 '15 at 14:12
  • V(i) seems to be the number of tokens of type (i). When v(i) is devided by N (total number of tokens) it yields an estimation for the probability of the type (i) [p(i) = V(i) / N]. – Claude Sep 4 '15 at 10:21

The entropy formula as quoted has some ideosyncrasies making it different from standard Shannon entropy

  • There is a (not really relevant) factor of 100 (probably to produce more beautiful numbers)
  • The term log N in the denominator is absent in Shannon entropy

The term p(i) cannot mean anything but the frequency of the i-th type, obtained by dividing the number of occurences of it by the total number of tokens. The frequency equals the probability in a bag-of-words (or more general, a bag-of-types) model.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.