I want to compare two corpora. One with 15000 tokens and the other with 5000 tokens. Can I normalise the word frequencies per 100000 words? Or I have to normalise based on words per million?
-
1I asked a lower level question on Stack Overflow recently with the reason being I wanted to do something very like this. I'll find a link for you, one answer seemed quite good. Here: Comparing frequency data and zipf / rank data– hippietrailCommented May 25, 2014 at 11:01
-
I don't think you have given enough information to make a judgement. If you were to stay within the same domain, you can possibly make such a judgement for the most commonly seen words, if you see that their frequencies converge as you increase the number of words. For low-frequency words, I don't see how you can extrapolate from 5K words to 1M words.– prash ♦Commented May 26, 2014 at 0:03
-
Well you can't expect "perfection" anyway. In fact you couldn't even really define what "perfect" could mean for such a scenario since every corpus is different. The only general principle really is that more data is better. As long as you're aware of the limitations I can think of many applications where some kind of useful results can be obtained.– hippietrailCommented May 26, 2014 at 12:15
Add a comment
|
1 Answer
Whether you normalise by 100,000 or 1,000,000 words is essentially a question of style and/or common practice in the journal or sub-field you're in. Mathematically, it makes not difference at all whether you say feature x occurs with a frequency of 10 per 100,000 words or 100 per 1,000,000 words. But you should be consistent, so if you provide other normalised frequencies or compare your results with previous findings, use the same normalisation index. In my experience, most corpus linguists give relative frequencies per million words, this is usually abbreviated as pmw. Some authors prefer normalisation per 100,000 words and call this the Mossé index, but this is in my experience relatively rare.
The only reason you might want to normalise by a smaller figure (e.g. per 1,000 words) is that some readers might interpret a frequency pmw as suggesting that you actually had a corpus of 1 M. words, not a (relatively small) 5,000 word corpus. So if you give no other normalised figures (from your own or other people's studies), it might be preferable to normalise by 1,000 words - especially if your audience is not very well acquainted with corpus linguistic methods.
-
Thank you so much for your nice comments. I was wondering if we have to use some statistical tests (chi square or log liklihood) to report on the differences of frequencies of words why should we do normalization at all?– user3650Commented May 27, 2014 at 18:18
-
You do both: Report normalised frequencies to show (in a way that is easy to understand) how frequent words A and B are, and statistical tests to determine whether word A is significantly more frequent than B (as opposed to a difference that is not significant).– robertCommented May 27, 2014 at 21:57