1

I have a corpus consisting of tweets by men and another corpus of female tweets. I was thinking of using a word embeddings approach (e.g., glove and fasttext) and examine the cosine similarity between a pre-defined set of paired words (e.g., work and status; work and support).

I would like to test the hypothesis that men are more likely than women to associate work with status. I was hoping to operationalize this by testing whether cosine similarity between work and status in men's tweets is statistically significantly higher than that between the same pair of words in women's tweets. Is it possible to check for statistical significance?

Let us say we get a cosine similarity score of 0.45 for men's corpus and 0.40 for women's corpus for work and status. How do I test that the difference of 0.05 is statistically significant at say p < 0.01?

Improve this question
1

This is not an answer to your question, but I think you need to re-examine your assumption: word vector similarity does not mean conceptual association, but instead "do these two words occur in similar contexts?"

For example, I took top 500k words from GloVe 840B data (300 dim) from here, and tried to find 20 closest words to work using cosine similarity:

 1 work         1.000000
 2 working      0.866403
 3 works        0.834335
 4 worked       0.764419
 5 done         0.719101
 6 well         0.688964
 7 doing        0.668836
 8 job          0.660955
 9 needed       0.655303
10 but          0.639304
11 much         0.629863
12 way          0.626623
13 so           0.624697
14 really       0.623545
15 time         0.622452
16 how          0.622418
17 able         0.622250
18 better       0.616725
19 need         0.616229
20 good         0.613265

As you can see, it has little to do with the concept of work, but rather a collection of common words that may be appear together (e.g., "Does it really work better?")

A more interesting example might be democracy:

 1 democracy    1.000000
 2 democratic   0.893857
 3 democracies  0.754781
 4 Democracy    0.714338
 5 socialism    0.713453
 6 capitalism   0.701732
 7 political    0.695850
 8 dictatorship 0.693884
 9 politics     0.689959
10 freedom      0.685923
11 communism    0.682284
12 freedoms     0.679934
13 ideology     0.674064
14 tyranny      0.672013
15 liberalism   0.658101
16 socialist    0.650724
17 pluralism    0.648239
18 independence 0.645715
19 equality     0.641852
20 constitution 0.638677

We see dictatorship, communism, and tyranny, because these words appear in the same context when democracy may be talked about. Further down, vote is only #438 (0.433892), lower than authoritarianism (#57), hegemony (#87), or apartheid (#213). But if you show these four words to people and ask "Which one is related to democracy?" then I feel most people will pick vote.

In conclusion, you can't use word vector to answer "Do people associate 'work' with 'status'?" At best, you can ask "Do people use 'work' and 'status' together in sentences?", which is not the same thing. (Also, I have a feeling that a lot of these tweets could be something like "This app's status bar refuses to work, and their support line is a joke!" - These are very versatile words.)

Improve this answer
4
  • This is a great explanation and helps me better conceptualize/articulate the analytical choice of word embeddings. Yes, I understand that word embeddings are more about "whether the words occur in similar context". My hypothesis is that women are less likely than men to associate work with power/status. I am assuming that language reflects pyschology. So, if men use work (and related words) and power/status (and related words) more often in the same tweet than women, then word embeddings and cosine similarity should capture it. Am I not right?
    – SanMelkote
    Jun 7 '20 at 8:32
  • 1
    Well, I don't know anything about computational linguistics, so I don't have much insight, but in my opinion - if that is your hypothesis, why don't you test it directly by counting the number of tweets that contain "work", "status", or both? From your description, I'm not sure what value is added by going through an additional step of word vectors.
    – jick
    Jun 7 '20 at 20:55
  • 1
    An additional benefit would be that you'll be able to directly sample, e.g., "20 random male tweets that contain both 'work' and 'status'" and verify whether they support your hypothesis. It's harder to do that with word vectors.
    – jick
    Jun 7 '20 at 20:56
  • The problem with a count-based approach is to come with a dictionary of all synonyms and terms that relate to status and work. A word-embeddings approach, in my understanding, can short-circuit this "dictionary-building" step.
    – SanMelkote
    Jun 7 '20 at 22:22
0

At first sight, it seems that the answer is a plain no: You just have two numbers, and talking about statistically significant differences of two numbers makes no sense at all.

But

The two numbers come from a process of measurement with a rather complex measuring apparatus (glove or fasttext). The process of measurement is not deterministic, but involves some random initialisations. So repeating the process of measurements with different random seeds leads to a distribution of measured values that can (hopefully!) be modelled by a one-dimensional Gaussian with a mean value and a variance. And now you can say that the difference of the values is significant (or not) with respect to this kind of variation. In analogy to a physical measurement, I will call the kind of error resulting in a variance Ablesefehler.

But, there is still more

Your measured values are measured on a samples of male and female tweets. A different sample will give different values, for sure. So you want to determine the sampling error in addition to the Ablesefehler. To determine the sampling error you need either more data or you have to divide your sample in subsamples and determine you cosine similarity for each subsample. Again, you get a (hopefully) one-dimensional Gaussian with mean and variance. Again, you can now say whether an observed difference is significant or not.

Note that for smaller samples the Ablesefehler may grow, determine it at sample size.

TL;DR It is possible to determine whether the difference is statistically significant by taking into account Ablesefehler and sampling error

Improve this answer
3
  • Many thanks for the detailed response. I need to grasp this in entirety and explore how to implement this.
    – SanMelkote
    Jun 6 '20 at 22:03
  • The arxiv.org/pdf/1608.07187.pdf paper suggests a word embeddings association test (WEAT) and WEFAT. I am not clear about how it works and its validity.
    – SanMelkote
    Jun 6 '20 at 22:23
  • 1
    Well, this seems like a good guideline: Don't use a suggested method unless you are clear about it and its validity.
    – jk - Reinstate Monica
    Jun 7 '20 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.