Q: A word like the name "Barry" might be very common in one of the corpus files (say a novel) and this will result in a larger than expected frequency for this word if you simply add all of its occurrences in the corpus and divide my 7 million.
Many corpora (except very large ones) only include parts of larger texts like novels (such as 2,000 words) to circumvent this problem. If, however, you have to use a corpus where such imbalances occur there is a way to address this problem.
- If you want to estimate the frequency of a word type you could give two normalised frequencies. An uncorrected frequency, and a corrected frequency that excludes tokens found in texts where the word on question is very frequent. For example, if the uncorrected frequency of work in the corpus is 50 per million words (pmw) you could exclude all texts where work is more than five times as frequent (more than 250 pmw) and calculate the corrected frequency based on the remaining texts in the corpus.
- If you want to compare the frequency of a word in two corpora the cut-off criterion should be the same for both corpora. It could for example be 5 times the average frequency of the word in both corpora. In any case you need to validate claims that a word is more frequent in one corpus by using a statistical test such as the log likelihood test.
Q: Words under a certain frequency should be ignored but how do I establish this threshold frequency? For example, if the word "Cranberry" occurs twice in the entire corpus, I can hardly claim that its frequency is 2/7000000.
Yes, you can, and in fact that is the only thing you can claim about the frequency of the word in the corpus. A different question is whether the word in question can be claimed to be more or less frequent in corpus A than in corpus B. That can be tested using the log likelihood test.
You could also ask
Given a frequency x of a word in a corpus, what is the likelihood that the true frequency of the word in the population is y?
This could be operationalised by imagining that you compile another corpus (with texts from the same registers!). What is then the likelihood that in the new corpus the frequency of the word is y?
Let's say in corpus x the word has a frequency of 2 pmw and you want to know how likely it is that in the population it is 20 pmw. Assuming your first corpus has 1,000,000 words, we imagine that you compile another corpus of 1,000,000 words and you find the word in question 20 times in that corpus. Using the log likelihood calculator, you get a log likelihood (also called G2) of 17.09. If you scroll down on the webpage you find a key for the log likelihood values, which tells you
99.99th percentile; 0.01% level; p < 0.0001; critical value = 15.13
Your G2 of 17.09 is higher than the critical value of 15.13, i.e. it is extremely unlikely that in the population the word occurs with a frequency of 20 pmw. In fact, there is a likelihood greater than 0.95 that the word is not more frequent than 8 pmw in the population (test this by putting 8 instead of 20 for the second corpus in the form).
