9

Sorry if my question is a little more mathematical in nature, but my question is:

Suppose I took a document of some length whether it be news article, book, or something of that sort. What sort of relationship would I expect between the document's length and the number of unique words contained in it.

There are two constraints a graph would contain: the line y = x meaning every word I read is unique and the line y = |english| where we recognize that English has a finite number of words, or some sort of upper bound although I don't want to get into a discussion on how many words are in the English language.

Perhaps more practically, how long a document on average would I need to retrieve to get a list of 100 unique English words, 1000 unique English words , or some other value? This of course would differ between languages, but I am interested in English.

Improve this question
5
  • There is no relationship (beyond the fact that the first page or so is limited by having so few words). A document is built of words in the writer's vocabulary, which is finite. A document generally has a purpose (or topic), and it's content will be limited to words relating to that purpose. You could have your printer print 3000 copies of a page with the same word repeated on it, and get 1 unique word across 3000 pages, or you could pick up a dictionary and have thousands of unique words across hundreds of pages.
    – Jim
    Dec 10, 2012 at 21:24
  • 7
    I think you're looking for Heaps' law. It basically says that as the corpus size increases, the number of unique words increases sublinearly. I'm sure people who know computational linguistics have some hard numbers.
    – SáT
    Dec 10, 2012 at 21:34
  • @Jim I am talking about real documents such as a CNN writer might put together, not toy documents. As a document increases in length, the same words will not be repeated over and over again. Rather, a longer document would expound further on the given topic. Agreed some words would be more common, but there is a relationship.
    – demongolem
    Dec 10, 2012 at 21:37
  • @SáT - If this were on-topic, I would upvote you. Thanks, that looks good.
    – demongolem
    Dec 10, 2012 at 21:38
  • @demongolem I suggest you use real documents, not CNN, because a fifth-grade vocabulary is quite limiting. Isn’t that what they are told to use? Unless it has gone down since. Yes, I’m serious, not ragging.
    – tchrist
    Dec 10, 2012 at 22:40

1 Answer 1

Reset to default
10

Thanks for the comments on Heaps’ Law. I followed one of the links to a related law, Herdan’s Law.

I’ll quote Wikipedia here:

The rule is as follows: if V is the number of different words in the text, and n is the length of the text, then V will be proportional to n to the power β:

        V ∝ nᵝ

where β ranges from 0.5 to 1 depending on the text.

(The equation looks better in one of the stackexchange families that understands equation formatting. My apologies.)

If one takes the conservative β of 0.5, look for a corpus of 10000 words to get you a V proportional to 100 unique words. At β of 0.6, your corpus would be proportional to 2200 words.

=== EDIT: Added real numbers for "Green Eggs and Ham" and a couple of CNN articles.

I tested "Green Eggs and Ham" and a pair of CNN articles to find the number of unique words.

  1. Green Eggs and Ham : 791 words, 55 unique.
  2. Angry with Obama, GOP threatens political war next year : 1553 words, 573 unique.
  3. New voter rules announced ahead of Egypt referendum : 408 words, 235 unique.

The β for these three are 0.600, 0.864, and 0.908. For the second one, I used V = nᵝ and took the log of both sides to solve for β. So β = log(V) / log(n).

Testing was done in Java. Sam-I-Am counts as a single word, as does decision-making.

=== EDIT: Added graphic for "Green Eggs and Ham"

The dependent, x-axis shows the word in the corpus. The independent, y-axis shows the unique count.

Green Eggs word count vs unique

Improve this answer
13
  • If the law is named after Gustav Herdan, better check to see if it's been widely verified.
    – jlawler
    Dec 11, 2012 at 5:29
  • Your mathematics is wrong (10000^0.6 is about 251 not 2200) and in any case there is the key word "proportional" which means the statement is not about the number of different words in a given text but about how it changes when the text length changes.
    – Henry
    Dec 11, 2012 at 7:37
  • It is odd that the Wikipedia article for Herdan's Law says it is also known as Heaps' Law but the article for Heaps' Law gives different values for β (specifically, the page for Heaps' Law states 0.4-0.6 plus instead of being a proportionality, they include a linear factor K with a value from 10-100. Can anyone shed light on this inconsistency?
    – acattle
    Dec 11, 2012 at 10:23
  • This may work as some sort of very rough rule of thumb for your average prose text, but it is easy enough to come up with counter examples, like a poem being a repetition of two words over five lines.
    – Cerberus
    Dec 11, 2012 at 13:34
  • @Cerberus I certainly agree with this. It would seem that at the low end and the high end of the x axis there is great error and it is somewhere in the middle of the curve where one would get the better performance.
    – demongolem
    Dec 11, 2012 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.