2

What we have gotten about the expected per word entropy of random yet grammatical text is just some upper bound of the the expected per word entropy, because we have not found the exact way to compute the probability of words. The models we adopt to compute the probability of words is just approximate models like bi-gram,trigram,etc,since the precise model of language,say,the grammar has not been established. By entropy formula and models we adopt, we can estimate that the exact entropy is smaller than what we have gotten.

There are lots of the approximate results about entropy per word, if the algorithm is correct,and assumptions are approximately valid, one can choose the least one from them. That is it. One should not expect perfect answer in the near future, just focus on the correctly approximated least one that appears.It is not difficult, since so many search tools are available in the days. what is important is that we can just get some upper bound of it,and the best solution is to choose the correctly approximated least one.So I think anyone that claims his model is precise, is wrong.

Someone does not agree with what I say above without any reason, I am wondering, if anyone can give any evidence to refute what I say above?

Also, Tarski's theorem on truth which can be deduced from Godel's incomplete theorem, the related theorems, have led to model theory, which is the fundamental part of semantics.The definition of pragmatics is somehow the third part of linguistics or semiotics, it is somehow what can not be incorporated into grammar and semantics,Shannon's classic article, has definitely claims that information does not considered any meaning of symbols, which excludes semantics and pragmatics. So we can conclude,any statement that claims to obtain precise information per word of a language by classical information applied to "semantics" and "pragmatics", is misunderstanding and misapplication, and serious researchers have to carefully know the difference among them.

The question is now, any evidence or valid theory can refute the relation among information theory, semantics, and pragmatics I say above?

Thanks in advance for any evidence, or reference that can refute all I say above. If there is controversy over any definition or theorem about information theory, I will post the question on other professional forum relating to information theory.

1

1 Answer 1

1

There are several points here that I should address.

What we have gotten about the expected per word entropy of random yet grammatical text is just some upper bound of the the expected per word entropy, because we have not found the exact way to compute the probability of words.

I agree (roughly) with the first part of the sentence; we are getting closer and closer to the upper bound of the expected per-word entropy. I disagree with the second part because there can't be an "exact way": entropy figures can be exact if were're dealing with a closed-world system, but when it comes to text, no matter how large the corpus, new text is usually bound to contain words never seen before. A sentence like "pass me the SATA cable" would never have been uttered a couple of decades ago, but are fairly common now.

The models we adopt to compute the probability of words is just approximate models like bi-gram,trigram,etc,since the precise model of language,say,the grammar has not been established.

Bigrams, trigrams, N-grams are all models of language, just as grammar models used in parser are. As before, there is no point looking for a "precise" model because the real-world language is best handled with an open-world model. But my point here is that there are plenty of grammar models. And some of them could be quite usable for deriving entropy figures for text. Table 1 of Learning Accurate, Compact, and Interpretable Tree Annotation shows the different (specialized) classes of words that the algorithm has induced. NNP-14 are months, NNP-4 and NNP-5 are common terminal words of names of corporations, etc. So, an IN-5 is rather more likely to be followed by something that ends with NNP-4, and an IN-2 is more likely to be followed by NNP-14, and so on. After all, such grammar models contain the probabilities of the co-occurrences of words with these POS tags.

There are lots of the approximate results about entropy per word, if the algorithm is correct,and assumptions are approximately valid, one can choose the least one from them.

I don't think it's helpful to think of algorithms as being "correct" or "incorrect". Is the grammar model used in the Stanford Parser "correct" because it can parse many sentences perfectly, or "incorrect" because it parses many sentences incorrectly? I think it's more helpful to think in terms of degree of suitability of each algorithm for a particular task.

One should not expect perfect answer in the near future...

I don't know what "perfect" would entail in this context.

So I think anyone that claims his model is precise, is wrong.

But no one who made such models claimed that their model is precise. As far as I know.

Shannon's classic article, has definitely claims that information does not considered any meaning of symbols, which excludes semantics and pragmatics.

Yes, Shannon's model for entropy has no concept of meaning or even sequence. Each symbol (letter or word, in our context) is assumed to occur completely independent of another symbol.

Shannon's definition of entropy

The above equation has no conception of sequence. Later models that don't ignore the sequence, such as n-gram models and Markov models have shown significantly better entropy figures.

So we can conclude,any statement that claims to obtain precise information per word of a language by classical information applied to "semantics" and "pragmatics", is misunderstanding and misapplication, and serious researchers have to carefully know the difference among them. The question is now, any evidence or valid theory can refute the relation among information theory, semantics, and pragmatics I say above?

I hope from all that I have explained above, it's clear that Shannon's definition of entropy was the beginning of the field of information theory, and not the end of it. In short, Shannon entropy sets a lower bound for a random sequence of symbols, but we know natural language is decidedly not a jumble of words. With better models, we get lower entropy figures.

"Semantics" does not have just one, static meaning. For the context of entropy, we could use distributional semantics to predict the next word or phrase in the sequence "pass me the ______". Ngram models would only predict the list of objects that they have previously seen in the context of these words in their training corpus. Models based on word meaning in context could do better and predict a candidate from all the words or phrases that denote tangible objects that can be held with one's hand, even if they had not been seen in the context of "pass me" before.

Models based on Frame semantics could use the context of the utterance "pass me the ______" to predict the missing word. In the context of a garage, it could be tools like hammers, screwdrivers, etc; in the context of a dining room, it could be salt, pepper, spoon, etc.

7
  • Why to leave out and not comment on "Also, Tarski's theorem on truth which can be deduced from Godel's incomplete theorem, the related theorems, have led to model theory, which is the fundamental part of semantics.The definition of pragmatics is somehow the third part of linguistics or semiotics, it is somehow what can not be incorporated into grammar and semantics,Shannon's classic article, has definitely claims that information does not considered any meaning of symbols, which excludes semantics and pragmatics.". This is most important. Nov 15, 2014 at 3:22
  • "The above equation has no conception of sequence. Later models that don't ignore the sequence, such as n-gram models and Markov models have shown significantly better entropy figures.". Have you read Shannon's article thoroughly? Nov 15, 2014 at 3:23
  • 1
    @XL_at_China for the first comment: I addressed that point: "Semantics" does not have just one, static meaning... For the second comment: No, I've not read Shannon's papers. I've only read newer books. If there's something you want me to address based on Shannon's papers, please modify your question and include that information. Please also cite papers (and provide links) in your questions, like I have, in my answer.
    – prash
    Nov 15, 2014 at 4:06
  • What are your definitions of grammar, semantics, and pragmatics? Have you read Tarski's paper on truth, and related theorems like Godel's incompleteness theorem? Have you known the outline of modern history of study of semantics and pragmatics? What are the boundaries among them? Nov 15, 2014 at 4:32
  • 1
    @XL_at_China Sorry, questions go above, in the question section.
    – prash
    Nov 15, 2014 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.