6

I have a English language corpus consisting of 100 files of different sizes and containing a total of 7 million words.

What is a good approach in order to establish the frequency of a given word from this corpus?

Issues:

  • A word like the name "Barry" might be very common in one of the corpus files (say a novel) and this will result in a larger than expected frequency for this word if you simply add all of its occurrences in the corpus and divide my 7 million.

  • Words under a certain frequency should be ignored but how do I establish this threshold frequency? For example, if the word "Cranberry" occurs twice in the entire corpus, I can hardly claim that its frequency is 2/7000000.

2
  • 1
    Brysbaert and New (2009) have a good recent discussion of issues that are relevant to your questions, including how much benefit there is to using contextual diversity (how many different documents a word occurs in) rather than raw counts, and the benefit of ignoring capitalized words which are probably proper names in a document -- not a big issue for Barry, but not doing it could seriously mess up your frequencies for common nouns like drake, grace, or rose. Sep 20, 2013 at 17:40
  • Even though you already received a good answer, I'd like to point out Gries' 2008 paper "Dispersions and adjusted frequencies in corpora" which is sort of a must-read for anyone doing corpora linguistics. It's available at stgries.info/research/2008_STG_Dispersion_IJCL.pdf.
    – Sixtyfive
    Jul 20, 2015 at 9:47

1 Answer 1

3

Q: A word like the name "Barry" might be very common in one of the corpus files (say a novel) and this will result in a larger than expected frequency for this word if you simply add all of its occurrences in the corpus and divide my 7 million.

Many corpora (except very large ones) only include parts of larger texts like novels (such as 2,000 words) to circumvent this problem. If, however, you have to use a corpus where such imbalances occur there is a way to address this problem.

  • If you want to estimate the frequency of a word type you could give two normalised frequencies. An uncorrected frequency, and a corrected frequency that excludes tokens found in texts where the word on question is very frequent. For example, if the uncorrected frequency of work in the corpus is 50 per million words (pmw) you could exclude all texts where work is more than five times as frequent (more than 250 pmw) and calculate the corrected frequency based on the remaining texts in the corpus.
  • If you want to compare the frequency of a word in two corpora the cut-off criterion should be the same for both corpora. It could for example be 5 times the average frequency of the word in both corpora. In any case you need to validate claims that a word is more frequent in one corpus by using a statistical test such as the log likelihood test.

Q: Words under a certain frequency should be ignored but how do I establish this threshold frequency? For example, if the word "Cranberry" occurs twice in the entire corpus, I can hardly claim that its frequency is 2/7000000.

Yes, you can, and in fact that is the only thing you can claim about the frequency of the word in the corpus. A different question is whether the word in question can be claimed to be more or less frequent in corpus A than in corpus B. That can be tested using the log likelihood test.

You could also ask

Given a frequency x of a word in a corpus, what is the likelihood that the true frequency of the word in the population is y?

This could be operationalised by imagining that you compile another corpus (with texts from the same registers!). What is then the likelihood that in the new corpus the frequency of the word is y?

Let's say in corpus x the word has a frequency of 2 pmw and you want to know how likely it is that in the population it is 20 pmw. Assuming your first corpus has 1,000,000 words, we imagine that you compile another corpus of 1,000,000 words and you find the word in question 20 times in that corpus. Using the log likelihood calculator, you get a log likelihood (also called G2) of 17.09. If you scroll down on the webpage you find a key for the log likelihood values, which tells you

99.99th percentile; 0.01% level; p < 0.0001; critical value = 15.13 

Your G2 of 17.09 is higher than the critical value of 15.13, i.e. it is extremely unlikely that in the population the word occurs with a frequency of 20 pmw. In fact, there is a likelihood greater than 0.95 that the word is not more frequent than 8 pmw in the population (test this by putting 8 instead of 20 for the second corpus in the form).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.