I'm looking for a robust way to roughly quantify the amount of information conveyed in a sentence, specifically in English. For instance "He went to the place" conveys less information than "John went to the store at 3 AM." Is there a standard way to do that? Would simply counting the number of content words be a robust enough approach?


First, the term depth is highly misleading here. Hearing the word depth will invoke the picture of a syntax tree in the mind of a trained linguist, and the depth of a tree is easily measured in terms of levels from the root.

But the question itself is sufficiently clear. To get at the information contained in a sentence (measured in bits) you need a language model. A simple language model is the so-called unigram model that measures the probability of a word just by taking its frequency in a suitable and large corpus. Summing up the terms $p\log p$ for the words in the sentence gives a measure of the information contained in the sentence.

Just counting "content words" is probably too crude, since nouns carry a lot more information than content verbs (in English language).


Ideally, the semantic content of an analysis should be language-agnostic. The semantic depth (informally speaking) should correspond to the complexity of the interaction between all the entities and the verbs (at least).

The best method for your scenario might turn out to be based entirely off named entity recognition and verbs in a sentence. To quote an example from spaCy documentation,

Apple|ORG is looking at buying U.K.|GPE startup for $1 billion|MONEY

There are more rigorous ways of quantifying the semantic complexity of sentences. To my knowledge, Discourse Representation Structures, is one of the most fundamental ones. There are more recent variants of the original DRS, which you might have to look into.

Here is an illustrative example from the paper by Liu, et al, (2018) Discourse Representation Structure Parsing:

each of the dead men wore
magazine vests and carried two hand grenades

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.