# How can I calculate if the difference between two word frequencies in one corpus is significant?

I want to study orthographical variants, for example:

Can firefighter and fire-fighter be considered orthographical variants (i.e. the difference in frequency in a corpus is not statistically significant), or should the one orthographical form be considered marked/irregular (i.e. the difference in frequency is statistically significant)?

In a corpus of 926,766,504 words, I get the following frequency counts (on lemmas):

• firefighter = 3,349 vs. fire-fighter = 336 vs. fire fighter = 1,338
• fireplace = 12,133 vs. fire-place = 95 vs. fire place = 1,369

What statistical measure(s) can I use to say, for example, that there is not a large enough difference between firefighter and fire-fighter to consider the latter marked, but that the difference between fireplace and fire-place is significant?

Two specific sub-questions:

1. If I work with probabilities (e.g. firefighter = 0,667; fire-fighter = 0,067; fire fighter = 0,266), is there a way to measure if the difference is significant? (Beyond just stating the obvious that the one occurs with a higher probability than the other.)
2. I'm new to log-likelihood, so sorry for a stupid question: Can I calculate the log-likelihood and effect size of a word-form within the same corpus? The online calculator of Paul Rayson seems to suggest that it needs to be from different corpora, but I would like to know if it can be within the same corpus as well?

• Although I regard this question as perfectly on-topic for this site, since there are unfortunately few computational linguistics here, you might get more answers on Cross Validated. – lemontree Nov 27 '16 at 18:17
• It seems that you're looking for practical significance, not statistical significance. I haven't done any calculations, but it's obvious from the numbers that the difference between the three forms of firefighter IS statistically significant, in that the difference is not purely due to chance. What kind of difference there has to be in order to call a form 'irregular' is not captured by statistical significance. Thus, general knowledge of statistics seems to be of little use here; the specialised knowledge of corpus linguists needs to be invoked. – WavesWashSands Nov 27 '16 at 18:21
• 'The online calculator of Paul Rayson seems to suggest that it needs to be from different corpora, but I would like to know if it can be within the same corpus as well?' - I think you need to be more specific about what you intend to do with it... – WavesWashSands Nov 27 '16 at 18:22
• For what it's worth: From a frequentist perspective, perhaps you could state a null hypothesis (where p, the population proportion - i.e. the ratio of a certain form's count against the total count - belongs to a set omega_0 where you would consider a form 'regular') and an alternative hypothesis (where p belongs to a set omega_1 where you'd consider a form 'irregular'), then get a generalised likelihood ratio test from there. (Not familiar with corpus linguistics though, so I'm not sure how well this'll work...) – WavesWashSands Nov 27 '16 at 18:31

(Diclaimer: I'm far from an expert in this area, but in my PhD coursework I took a computational linguistics course and a stats for linguistics course.)

This seems like a case of a single categorical dependent variable. For this we used a Pearson chi-squared test to see if distribution differs from what is expected. This requires you to have an expectation. If you don't have one to start out with, equal distribution (or null hypothesis) is a good place to start.

## Hypothesis #1 - Equal distribution

So let's hypothesize that these three orthographic forms are in free variation and we have no reason to assume any of them is more likely than the other. So we expect each variant form to account for 1/3 of all occurrences. It's easy to calculate the significance in R (with RStudio).

For firefighter:

``````> FFs = c(3349, 336, 1338) # the observed distribution
> names(FFs)<-c("firefighter", "fire-fighter", "fire fighter") # add labels
> chisq.test(FFs, p=c(1/3, 1/3, 1/3)) # expecting equal distribution

Chi-squared test for given probabilities

data:  FFs
X-squared = 2812.3, df = 2, p-value < 2.2e-16
``````

And for fireplace:

``````> FPs = c(12133, 95, 1369)
> names(FPs)<-c("fireplace", "fire-place", "fire place")
> chisq.test(FPs, p=c(1/3, 1/3, 1/3))

Chi-squared test for given probabilities

data:  FPs
X-squared = 19298, df = 2, p-value < 2.2e-16
``````

These are huge effects. This data has 2 degrees of freedom (df = the number of categories - 1) and so a chi-squared score above 4.605 has statistical significance. A score of 13.8 has a p-value of 0.001. So this is huge.

But this was a very simplified test, and to really get at this problem, you'd probably want to take a more nuanced approach. You could collect data on several compounds that have hyphenated and spaced variations and look at the overall distribution to get a more realistic set of expected values (the `p=c(x,y,z)` argument in the function).

## Hypothesis #2 - Distributed similarly to fireplace, fire-place, fire place

As a one-step-forward example, we could use the distribution of fireplace terms as our expected distribution for firefighter.

``````> FFs.exp<-c(FPs/sum(FPs)) # this takes the count of each fireplace variant and divides it by the total number of occurrences
> chisq.test(FFs, p=FFs.exp)

Chi-squared test for given probabilities

data:  FFs
X-squared = 4236.1, df = 2, p-value < 2.2e-16
``````

So here too, the difference in distribution of the three variants is incredibly significant. What this test doesn't tell you is which form is marked. For that, you only need look at the raw frequencies compared to the expected frequencies.

``````> FFs
firefighter fire-fighter fire fighter
3349          336         1338
> c(sum(FFs)*FFs.exp)
fireplace fire-place fire place
4482.16952   35.09487  505.73560
``````

Here the hyphenated and spaced forms are more frequent than expected. So what this test is showing us is that the hyphenated and spaced forms are much less marked for firefighter than comparable forms for fireplace are.