5

I came across the concept of lexical density while reading "Embracing a New Creed: Lexical Patterning and the Encoding of Ideology"[1] by Oliver Mason and Rhiannon Platt and was wondering what practical benefit it is for linguistic analysis.

[1] Oliver Mason and Rhiannon Platt (2006) "Embracing a New Creed: Lexical Patterning and the Encoding of Ideology", College Literature, vol.33, no.2, 155—170.

edit

I've revised my understanding of lexical density based on re-reading the article in question and getting a clearer understanding of the actual calculation. I now understand lexical density to be the proportion of content words (nouns, main verbs, adjectives, and adverbs) to function words (articles, prepositions, conjunctions, auxiliary verbs, and pronouns). Essentially I guess I understand it as the proportion of words that give meaning to the constituent clauses and phrases over against the words that appear in nearly every context and lend little to no meaning.

6
  • 1
    For the sake of completeness, could you include a reference to the paper? Apr 9, 2013 at 18:19
  • The paper in question is available here. And it seems to define "lexical density" differently from what's said in the question, as: the ratio of lexical (or content) words to grammatical (or function) words. Apr 9, 2013 at 23:55
  • @GastonÜmlaut then it's probably my ignorance. Would you mind helping me understand what is actually being said. I'd hate to misrepresent.
    – swasheck
    Apr 10, 2013 at 2:16
  • Sorry, I didn't get back to this until just now, but your edit is good. Apr 10, 2013 at 11:13
  • 1
    Is lexical density supposed to vary according to context, or speaker, or (typological variety of) language?
    – jlawler
    May 5, 2014 at 18:34

2 Answers 2

2

I think Harald Baayen in his book Analyzing Linguistic Data: a practical introduction to statistics argues against the value of similar measurements quite eloquently. In Chapter 6.5:

If we read through a text or corpus, and at regular intervals keep note of how many different types we have encountered, we find that, unsurprisingly, the number of types increases, first rapidly, and then more and more slowly...[Growth Curve of the Vocabulary].... The vocabulary growth rate is estimated by the ratio of the number of hapax legomena to the number of tokens sampled. The growth rate is a probability, the probability that, after having read N tokens, the next token sampled represents from unseen type, a word type that did not occur among the preceding N tokens[Good, 1953, Baayen 2001]

He goes on to note that

The problem that arises is that, although we could select the total number of types counted for the full text as a measure of lexical richness, this measure would not lend itself well for comparison with longer or with shorter texts. Therefore considerable effort has been invested in developing measures of lexical richness that would supposedly be independent of the number of tokens sampled.The third panel on the upper row shows the worst measure of all, the type-token ratio, obtained by dividing the number of types by the number of tokens. It is highly correlated (r=0.99) with the growth rate of the vocabulary shown in the panel to its left...We return to this issue below, here we note that there is no sign that the curve is anywhere near reaching a stable value. None of these putative constants is a true constant. The only measure of these last four that is, at least under the simplifying assumption that words are used randomly and independently, truly constant is Yule's K...

(Page 244) I chopped some stuff out.

It would be interesting to see if lexical density was also unstable.

0

If we take a sample of supposedly analytical language, then the samples of highest density are likely to be grammatical clauses. The items of highest frequency are likely to be grammatical particles.

Taking this as a starting point, we can deduce the grammatical meaning of a phrase/paragraph/clause, and perhaps even the limits of a sentence for a language where it is marked grammatically (e.g. SOV or OSV-languages, or languages where aspect is marked by a final particle in a sentence. Yes, like in Chinese).

3
  • 1
    Why would that be? The formula counts tokens, not token combinations or syntactic constituents, so it couldn't even recognize clauses.
    – jlawler
    Apr 9, 2013 at 20:00
  • There should be a way for clause recognition by a formula. Or we could use the formula first, then our brains.
    – Manjusri
    Apr 9, 2013 at 20:02
  • 1
    Oh, there are definitely ways of recognizing clauses. But the process discussed here is not one of them. It operates on a lexical usage basis only, which is interesting; but its utility remains to be seen. In any event, the article is behind a pay wall, and is not published in a journal where linguistic analysts would likely notice it.
    – jlawler
    Apr 9, 2013 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.