There are several points here that I should address.
What we have gotten about the expected per word entropy of random yet grammatical text is just some upper bound of the the expected per word entropy, because we have not found the exact way to compute the probability of words.
I agree (roughly) with the first part of the sentence; we are getting closer and closer to the upper bound of the expected per-word entropy. I disagree with the second part because there can't be an "exact way": entropy figures can be exact if were're dealing with a closed-world system, but when it comes to text, no matter how large the corpus, new text is usually bound to contain words never seen before. A sentence like "pass me the SATA cable" would never have been uttered a couple of decades ago, but are fairly common now.
The models we adopt to compute the probability of words is just approximate models like bi-gram,trigram,etc,since the precise model of language,say,the grammar has not been established.
Bigrams, trigrams, N-grams are all models of language, just as grammar models used in parser are. As before, there is no point looking for a "precise" model because the real-world language is best handled with an open-world model. But my point here is that there are plenty of grammar models. And some of them could be quite usable for deriving entropy figures for text. Table 1 of Learning Accurate, Compact, and Interpretable Tree Annotation shows the different (specialized) classes of words that the algorithm has induced. NNP-14 are months, NNP-4 and NNP-5 are common terminal words of names of corporations, etc. So, an IN-5 is rather more likely to be followed by something that ends with NNP-4, and an IN-2 is more likely to be followed by NNP-14, and so on. After all, such grammar models contain the probabilities of the co-occurrences of words with these POS tags.
There are lots of the approximate results about entropy per word, if the algorithm is correct,and assumptions are approximately valid, one can choose the least one from them.
I don't think it's helpful to think of algorithms as being "correct" or "incorrect". Is the grammar model used in the Stanford Parser "correct" because it can parse many sentences perfectly, or "incorrect" because it parses many sentences incorrectly? I think it's more helpful to think in terms of degree of suitability of each algorithm for a particular task.
One should not expect perfect answer in the near future...
I don't know what "perfect" would entail in this context.
So I think anyone that claims his model is precise, is wrong.
But no one who made such models claimed that their model is precise. As far as I know.
Shannon's classic article, has definitely claims that information does not considered any meaning of symbols, which excludes semantics and pragmatics.
Yes, Shannon's model for entropy has no concept of meaning or even sequence. Each symbol (letter or word, in our context) is assumed to occur completely independent of another symbol.
The above equation has no conception of sequence. Later models that don't ignore the sequence, such as n-gram models and Markov models have shown significantly better entropy figures.
So we can conclude,any statement that claims to obtain precise information per word of a language by classical information applied to "semantics" and "pragmatics", is misunderstanding and misapplication, and serious researchers have to carefully know the difference among them.
The question is now, any evidence or valid theory can refute the relation among information theory, semantics, and pragmatics I say above?
I hope from all that I have explained above, it's clear that Shannon's definition of entropy was the beginning of the field of information theory, and not the end of it. In short, Shannon entropy sets a lower bound for a random sequence of symbols, but we know natural language is decidedly not a jumble of words. With better models, we get lower entropy figures.
"Semantics" does not have just one, static meaning. For the context of entropy, we could use distributional semantics to predict the next word or phrase in the sequence "pass me the ______". Ngram models would only predict the list of objects that they have previously seen in the context of these words in their training corpus. Models based on word meaning in context could do better and predict a candidate from all the words or phrases that denote tangible objects that can be held with one's hand, even if they had not been seen in the context of "pass me" before.
Models based on Frame semantics could use the context of the utterance "pass me the ______" to predict the missing word. In the context of a garage, it could be tools like hammers, screwdrivers, etc; in the context of a dining room, it could be salt, pepper, spoon, etc.
