How can I calculate if the difference between two word frequencies in one corpus is significant?

Question

I want to study orthographical variants, for example:

Can firefighter and fire-fighter be considered orthographical variants (i.e. the difference in frequency in a corpus is not statistically significant), or should the one orthographical form be considered marked/irregular (i.e. the difference in frequency is statistically significant)?

In a corpus of 926,766,504 words, I get the following frequency counts (on lemmas):

• firefighter = 3,349 vs. fire-fighter = 336 vs. fire fighter = 1,338
• fireplace = 12,133 vs. fire-place = 95 vs. fire place = 1,369

What statistical measure(s) can I use to say, for example, that there is not a large enough difference between firefighter and fire-fighter to consider the latter marked, but that the difference between fireplace and fire-place is significant?

Two specific sub-questions:

1. If I work with probabilities (e.g. firefighter = 0,667; fire-fighter = 0,067; fire fighter = 0,266), is there a way to measure if the difference is significant? (Beyond just stating the obvious that the one occurs with a higher probability than the other.)
2. I'm new to log-likelihood, so sorry for a stupid question: Can I calculate the log-likelihood and effect size of a word-form within the same corpus? The online calculator of Paul Rayson seems to suggest that it needs to be from different corpora, but I would like to know if it can be within the same corpus as well?

Thank you in advance for your help!

Answer 1

(Diclaimer: I'm far from an expert in this area, but in my PhD coursework I took a computational linguistics course and a stats for linguistics course.)

This seems like a case of a single categorical dependent variable. For this we used a Pearson chi-squared test to see if distribution differs from what is expected. This requires you to have an expectation. If you don't have one to start out with, equal distribution (or null hypothesis) is a good place to start.

Hypothesis #1 - Equal distribution

So let's hypothesize that these three orthographic forms are in free variation and we have no reason to assume any of them is more likely than the other. So we expect each variant form to account for 1/3 of all occurrences. It's easy to calculate the significance in R (with RStudio).

For firefighter:

> FFs = c(3349, 336, 1338) # the observed distribution
> names(FFs)<-c("firefighter", "fire-fighter", "fire fighter") # add labels
> chisq.test(FFs, p=c(1/3, 1/3, 1/3)) # expecting equal distribution

    Chi-squared test for given probabilities

data:  FFs
X-squared = 2812.3, df = 2, p-value < 2.2e-16

And for fireplace:

> FPs = c(12133, 95, 1369)
> names(FPs)<-c("fireplace", "fire-place", "fire place")
> chisq.test(FPs, p=c(1/3, 1/3, 1/3))

    Chi-squared test for given probabilities

data:  FPs
X-squared = 19298, df = 2, p-value < 2.2e-16

These are huge effects. This data has 2 degrees of freedom (df = the number of categories - 1) and so a chi-squared score above 4.605 has statistical significance. A score of 13.8 has a p-value of 0.001. So this is huge.

But this was a very simplified test, and to really get at this problem, you'd probably want to take a more nuanced approach. You could collect data on several compounds that have hyphenated and spaced variations and look at the overall distribution to get a more realistic set of expected values (the p=c(x,y,z) argument in the function).

Hypothesis #2 - Distributed similarly to fireplace, fire-place, fire place

As a one-step-forward example, we could use the distribution of fireplace terms as our expected distribution for firefighter.

> FFs.exp<-c(FPs/sum(FPs)) # this takes the count of each fireplace variant and divides it by the total number of occurrences
> chisq.test(FFs, p=FFs.exp)

    Chi-squared test for given probabilities

data:  FFs
X-squared = 4236.1, df = 2, p-value < 2.2e-16

So here too, the difference in distribution of the three variants is incredibly significant. What this test doesn't tell you is which form is marked. For that, you only need look at the raw frequencies compared to the expected frequencies.

> FFs
 firefighter fire-fighter fire fighter 
        3349          336         1338 
> c(sum(FFs)*FFs.exp)
 fireplace fire-place fire place 
4482.16952   35.09487  505.73560 

Here the hyphenated and spaced forms are more frequent than expected. So what this test is showing us is that the hyphenated and spaced forms are much less marked for firefighter than comparable forms for fireplace are.

Note about R

I realize this is a lot of code, and if you're not used to working with R, it's not the most intuitive programming language to read. But Stefan TH. Gries has two good books on linguistics with R if you're interested: Quantitative Corpus Linguistics with R and Statistics for Linguistics with R.

There are a lot of good Python resources out there too, if you'd like to work with a more human-readable programming language. But for statistics, R is really the best.