How can I know if a log-likelihood score is high enough?

Question

I am trying to understand if some terms can be considered as distinctly American or British. So for example I was studying the term "aubergine" and the normalized frequency in the American corpus is of 409 while in British ENglish it is of 3637. I normalized the frequency by making the larger corpus of the same size of the smaller one which is of 34 billion words (since it's my first time using corpora I hope that This is the right process to normalize frequencies: multiply the raw F. by the desired size of the corpus and later divide the end result by the real size of the curpus). Now, what I am trying to understand is if a LL of 2959 is high enough to say that the term aubergine is a key word in British English or not. In the book that I am studying the author uses a cut-off of 50 but since the corpora that I am using are much larger than his I would like to know what cut-off I should use. I am really sorry if I haven't made myself clear but I am very confused because as I said I never used corpora before and also English is not my first language, so I apologize in advance for any mistakes.

Answer 1

For the purpose of normalisation, you should use a formula such as this:

normalised_frequency_per_million_words = raw_frequency / corpus_size * 1,000,000

This formula can be adjusted to normalise, for example, per 100,000 words - just replace the 1,000,000 in the formula by 100,000.

When calculating log likelihood for a comparison of the frequency of a linguistic item (e.g. a word) in the two corpora, you should use the raw frequency of the linguistic item and the exact size of the corpora. Do NOT use the normalised figures in the formula. For the interpretation of log likelihood, Paul Rayson gives the following guidelines:

95th percentile; 5% level; p < 0.05; critical value = 3.84

99th percentile; 1% level; p < 0.01; critical value = 6.63

99.9th percentile; 0.1% level; p < 0.001; critical value = 10.83

99.99th percentile; 0.01% level; p < 0.0001; critical value = 15.13

This provides information on the statistical significance of the difference - basically the question of whether with a new corpus you would get a similar result. However, in order to decide if the difference is meaningful, you need to rely on effect size. Is the linguistic item 20% more frequent in corpus A than in corpus B? Or twice as frequent? Whether you consider 20% or even 5% meaningful or not is up to you to decide based on your research question. When comparing two varieties of a language I would not consider a 5% difference to be particularly meaningful.