5

I came across the concept of lexical density while reading "Embracing a New Creed: Lexical Patterning and the Encoding of Ideology"[1] by Oliver Mason and Rhiannon Platt and was wondering what practical benefit it is for linguistic analysis.

[1] Oliver Mason and Rhiannon Platt (2006) "Embracing a New Creed: Lexical Patterning and the Encoding of Ideology", College Literature, vol.33, no.2, 155—170.

edit

I've revised my understanding of lexical density based on re-reading the article in question and getting a clearer understanding of the actual calculation. I now understand lexical density to be the proportion of content words (nouns, main verbs, adjectives, and adverbs) to function words (articles, prepositions, conjunctions, auxiliary verbs, and pronouns). Essentially I guess I understand it as the proportion of words that give meaning to the constituent clauses and phrases over against the words that appear in nearly every context and lend little to no meaning.

6
  • 1
    For the sake of completeness, could you include a reference to the paper? – Otavio Macedo Apr 9 '13 at 18:19
  • The paper in question is available here. And it seems to define "lexical density" differently from what's said in the question, as: the ratio of lexical (or content) words to grammatical (or function) words. – Gaston Ümlaut Apr 9 '13 at 23:55
  • @GastonÜmlaut then it's probably my ignorance. Would you mind helping me understand what is actually being said. I'd hate to misrepresent. – swasheck Apr 10 '13 at 2:16
  • Sorry, I didn't get back to this until just now, but your edit is good. – Gaston Ümlaut Apr 10 '13 at 11:13
  • 1
    Is lexical density supposed to vary according to context, or speaker, or (typological variety of) language? – jlawler May 5 '14 at 18:34
2

I think Harald Baayen in his book Analyzing Linguistic Data: a practical introduction to statistics argues against the value of similar measurements quite eloquently. In Chapter 6.5:

If we read through a text or corpus, and at regular intervals keep note of how many different types we have encountered, we find that, unsurprisingly, the number of types increases, first rapidly, and then more and more slowly...[Growth Curve of the Vocabulary].... The vocabulary growth rate is estimated by the ratio of the number of hapax legomena to the number of tokens sampled. The growth rate is a probability, the probability that, after having read N tokens, the next token sampled represents from unseen type, a word type that did not occur among the preceding N tokens[Good, 1953, Baayen 2001]

He goes on to note that

The problem that arises is that, although we could select the total number of types counted for the full text as a measure of lexical richness, this measure would not lend itself well for comparison with longer or with shorter texts. Therefore considerable effort has been invested in developing measures of lexical richness that would supposedly be independent of the number of tokens sampled.The third panel on the upper row shows the worst measure of all, the type-token ratio, obtained by dividing the number of types by the number of tokens. It is highly correlated (r=0.99) with the growth rate of the vocabulary shown in the panel to its left...We return to this issue below, here we note that there is no sign that the curve is anywhere near reaching a stable value. None of these putative constants is a true constant. The only measure of these last four that is, at least under the simplifying assumption that words are used randomly and independently, truly constant is Yule's K...

(Page 244) I chopped some stuff out.

It would be interesting to see if lexical density was also unstable.

0

If we take a sample of supposedly analytical language, then the samples of highest density are likely to be grammatical clauses. The items of highest frequency are likely to be grammatical particles.

Taking this as a starting point, we can deduce the grammatical meaning of a phrase/paragraph/clause, and perhaps even the limits of a sentence for a language where it is marked grammatically (e.g. SOV or OSV-languages, or languages where aspect is marked by a final particle in a sentence. Yes, like in Chinese).

3
  • 1
    Why would that be? The formula counts tokens, not token combinations or syntactic constituents, so it couldn't even recognize clauses. – jlawler Apr 9 '13 at 20:00
  • There should be a way for clause recognition by a formula. Or we could use the formula first, then our brains. – Manjusri Apr 9 '13 at 20:02
  • 1
    Oh, there are definitely ways of recognizing clauses. But the process discussed here is not one of them. It operates on a lexical usage basis only, which is interesting; but its utility remains to be seen. In any event, the article is behind a pay wall, and is not published in a journal where linguistic analysts would likely notice it. – jlawler Apr 9 '13 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.