I am looking for a basic mathematical model of information encoded in a text.
For the sake of presentation I adopt a language learner perspective: how many words does one need to know, in order to understand a book (provided that one knows the basic grammar or at least morphology)? An example given here (the first installment of Harry Potter in Hebrew) contains the main features of the problem:
- The most frequent words are prepositions, conjunctions, auxiliary verbs, etc. that by themselves carry little information (or no information at all).
- Most of the words occur only once. These are clearly informative ones, but anyone who has ever learned a language knows that one does not need to know all the words to understand a book - in fact, at a certain level one learns new words by encountering them. Moreover, most second language speakers vocabulary remains a fraction of that of native speakers, without significantly hampering their ability to interact with natives.
These features are summarized by Zipf's law and/or rarefaction curves. I suppose these could be combined with the Shannon theory of information to relate the information content understood with the size of the reader's vocabulary.
I would particularly appreciate references to books or original articles.
