I am looking for a human understandable explanation of a famous feature of word vectors, namely the equation
|king⟩+|woman⟩-|man⟩≃|queen⟩
How does it come that this equation holds?
This question is inspired by the thread of comments to my answer here: https://linguistics.stackexchange.com/a/35836/9781
The basic mathematical idea behind word vectors is very simple: each word i has a "word vector" and (following the terminology of GloVe paper) another "context vector", which I'll denote by w[i] and c[i], respectively.
The word vector algorithms try to compute these vectors, with the goal being that: the dot product w[i] * c[j] is high if and only if word i occurs frequently near word j.
That's all. So the famous equation w[king]+w[woman]-w[man] ≃ w[queen] can be rewritten as: w[woman] - w[man] ≃ w[queen] - w[king]. OK, but then what is the left-hand side?
As I said, w[man] * c[i] is high if and only if word i occurs frequently around man. Hence, (w[woman] - w[man]) * c[i] = (w[woman] * c[i]) - (w[man] * c[i]) is high if and only if word i occurs more frequently around woman than around man.
Consider a set of words that have different probability to appear around man vs. woman, then the algorithm tries to align the vector v1 = w[woman] - w[man] such that:
If word i occurs more frequently around woman than man, then v1 is "dragged toward" the same direction as context vector c[i] (so that the dot product is high).
If word j occurs less frequently around woman than man, then v1 is "dragged toward" the opposite direction as context vector c[j] (so that the dot product is a large negative number).
That's the basic idea, and everything else in various papers are basically fine-tuning to do it more efficiently.
(Note that these vectors usually have hundreds of dimensions: so unlike a 3-D vector, a word vector can have "similar direction" as thousands of different vectors. High-dimensional spaces are weird.)
As a result, we can clearly see that v1 will align toward the same direction as c[she], c[her], c[pretty], or -c[he], -c[him], -c[muscular], etc.
But that's also (roughly) the same for v2 = w[queen] - w[king]!
Of course it doesn't align perfectly. (We don't say "pretty queen" that often, after all.) As another example, during the training, there must be a very strong force to align v2 with c[elizabeth], but it must be relatively weak for v1. (You may object that there being a famous Queen Elizabeth II has nothing to do with the innate meaning of queen: that would be correct and utterly irrelevant. Word vector doesn't care about innate meaning of words: it's all about what happens around these words in actual corpora.)
Similarly, there must be a force to align -v1 with various interjections ("Man, that sucks", "hey man"), which would be absent for v2.
But it's still a good enough match: remember, the matches are far from perfect. It just happens so that w[queen] is still the closest word vector from the expected point of w[king] + w[woman] - w[man].
One could imagine that you could model a word's semantics with an extremely long vector: each coordinate is a semantic feature and the value is +1, -1, or 0 for positive, negative, and don't care. So 'man' might for -all- the semantic features possible. It is straightforward to see that vector subtraction removes common features, and then vector addition adds in features, so that king - man might just be <..., ruler:+1, ....> and adding woman = <..., male:-1,... > would yield <... ruler:+1, male:-1...> which is the same as queen.
So these coordinates are very binary.
Word2vec, or GloVe, produces vectors for words which are much, much shorter, say only 300 coordinates. Think of these vectors as compressed versions of their full vectors we just discussed, and also the 300 coordinates don't really map to individual features directly. Think of it as a dimensionality reduction, like PCA but instead of starting off with known vectors for words, word2vec figures them out from context examples (via CBOW or skip gram). So the primary difficulty here then is whether the implicit compression mechanism from the full set of features to just 300 preserves distances.
Since we aren't really compressing the vector space, it really is just an engineering confirmation that when vectors are added and subtracted that they do indeed seem to not lose too much similarity in the smaller 300 feature version. It is important to note that the vector comparison isn't exact; '≃' means 'is nearby to' and 'nearby' really means 'is maybe a synonym or highly associated with'. You don't get |queen> from the vector operations, just a new vector that can be checked is very close to |queen>.
(as a side note, compressed semantic vectors for words can be produced from a PCA procedure and a confusion matrix or collocation matrix, but the quality of the vectors produced has been improved dramatically by using a neural network instead)
Edit: Here's a perhaps slightly more accessible version of the my long-winded original post: It's just vectors, right. I can create a little gender-nobility continuum and put some words on it like so:
+--------------+
| gender |
+----------+------+-------+
| | man | woman |
| nobility +------+-------+
| | king | queen |
+----------+------+-------+
my_vecs = open('my_vecs.txt','w+')
my_vecs.write('4 2\nman -1.0 -1.0\nwoman 1.0 -1.0\nking -1.0 1.0\nqueen 1.0 1.0')
my_vecs.close()
my_vecs = KeyedVectors.load_word2vec_format("my_vecs.txt")
results = my_vecs.most_similar(positive=['king','woman'],negative=['man'])
print(results)
# ('queen', 0.9999999403953552)]
Big surprise, right? So we can skip over "how does this work," because that's easy, and get right to the deeper question with regard to mainstream practices, "how do these words get coordinates such that the equation holds?" For this, look to the training methods, which vary, but are largely spatially relational in the sequence, as in relating words in sequential proximity and otherwise. Unfortunately, this doesn't build a space of meaning like the equation hopes for, but rather builds a space where words are related (varying slightly by methods) by the frequency that a word appears in proximity to another word. That's essentially all there is to it. You can look at my code examples below to see it in action.
--- original post begins ---
As you surely recall from our previous conversations, networks produce the results you've described precisely because they were designed to, which is generally to combat all forms of ambiguity in language modeling, or, said another way, to preserve more information than can be preserved by mere tokenized word sequences. An example application objective would be to extract some information as pertains to some understanding---and I stress 'understanding' here, in that we're involving the concept of meaning from the very outset---from some sequence of text. For example, probability of spam as a function of email content, or a person's political alignment as a function of the contents of their tweets. General practices involve tokenizing words according to some criteria (e.g. order or frequency of occurrence, etc), which would be fine if words and their orders had precisely one meaning, but that's clearly a preposterous expectation of human language; not only do words have multiple (and frequently very unrelated) meanings, syntax can vary wildly and even carry meaning itself! There are many reasons why quantizing language makes for difficult comprehension and modeling. After all, it's essentially setting out to model a continuum by first quantizing all your information.
Thankfully, topological semiotics can ameliorate this. In great brevity, there are two key concepts relevant to this discussion:
- An ideal simulacrum of the physically real is as continuous as physical reality.
- "Comprehensible space" (a manifold of aggregated interpretants) receives novel input only as differentials.
The first of these, as pertains to this explanation, simply indicates (borrowing from Charles Sanders Peirce's triadic model) that an interpretant (a subjective experiential understanding of reality, if you will) should be as continuous as the object whose impressions became the signals that instigated it. Relating this to some of the aforementioned problems, consider that the meaning of the word "under" is not (in any realistically comprehensible way) related to the meanings of its constituent signs (e.g. letters), just as the meaning of "under the weather" is scarcely relatable to the meaning of its constituent signs (e.g. collocations, words, letters, and so-on); understanding the meaning of this idiom depends on knowledge of both human interaction with storms (e.g. to know that one might become ill), and an understanding of the human experience of illness (to know that this is generally undesirable). Attempting to quantize this continuous nature as a hierarchy as we tend to attempt (e.g. moments ago when I mentioned constituent signs) is both unnecessary because we can model meaning continuously, and futile because hierarchies are themselves constructs. In simpler terms: manifold learning is an ideal choice for simulating relative sign meanings.
The second above concept may seem strange and unrelated, but it carries several critical implications, of which the following is most pertinent: what is known can only exist relative to what has been known. In the more elegant words of Roland Barthes, "No sooner is a form seen than it must resemble something: humanity seems doomed to analogy." This permits imagination, but confines understanding to the space of that which has been previously experienced. In other words, experiences of reality can only exist relative to themselves; our model of language meanings can only describe meaning relative to that from which its landscape was shaped. In our application, the transformation we end up with (i.e. the features of the network), which typically receives tokenized sequences and returns vector representations within the manifold of our designing, can only provide meanings relative to the corpus on which it was trained (and, indeed, the route of navigation through that corpus), varying in depiction---which is to say, varying in the way that it describes meaning---by the method of modeling. For example, the "skipgram" model describes meaning as spatially relational context (meaning points to context), while the "continuous bag of words" model describes meaning as consisting of spatially relational context (context points to meaning).
There are obviously some heavy assumptions being made here, and not exclusively good ones. We know that relative frequency of relative sequential word position doesn't truly carry all the meanings that can be crafted into a sequence. This should come as no surprise, of course, since we're attempting to quantize a continuous relationship; creating a discrete manifold of understanding to describe continuous relationships. Shame on us, but, as you can see, it's a difficult habit to break. Nevertheless, the key take-away here is that the primary objective described above, regardless of which method you use to generate your model, is to find an equation that transforms the vector representations of tokenized sequences into vector representations of relative meanings---or, at least, the best simulacrum that a particular corpus, technique, and architecture can provide. As before, what a particular axis (or dimension) represents varies by method, and can be as arbitrary as x, y and z, or quite specific. For example, if your purposes can afford a softmax activation function, you can describe vector representations as relative constituency, and that's amusingly elegant: you could describe everything as pertains to its relationship with the words "man," "bear," and "pig," for which the mythological "man-bear-pig" might dwell somewhere in the midst. For better understanding, we can observe the same action in reverse: the secondly mentioned concept of topological semiotics indicates that an understanding of a "man-bear-pig" depends solely on understanding(s) of "man," "bear," "pig," and nothing more. As predicted, training with a softmax activation function, which is a constrained topology, indeed requires precisely that!
In terms perhaps more familiar to the linguistically inclined, consider this alternative depiction: the word "man" can produce ample interpretants, especially since the nature of interpretants should be expected to be, as aforementioned, pretty continuous. For example, the word "queen" could be used in reference to a monarch, or to a suit of playing cards, or to a person bearing such a name, among other things. Meanwhile, a queen (monarch) of the lineage "Queen" could appear more or less similar to a queen (playing card); did Lewis Carroll not evoke precisely this depiction? We can make our models high-dimensional to ameliorate the quantization inherent in dimensionality (much as how increasing the number of edges of a polygon better simulates a circle), giving more freedom for relational complexity: "man" and "woman" can reside simultaneously near to each other along some axes (e.g. such that a region might resemble "species") and distant along others (e.g. such that a region might resemble "gender"). Thankfully, we're able to understand our transforming from sign to interpretant (and so-on) because these operations are entirely self-supervised, and which is the action of understanding the meaning of what you're reading. So, then, if I ask you for a word with a meaning most closely resembling that of "big" in the phrase "a big pizza," you can consider the meaning of "big" as pertains to the given sentence, and find something very close to it (literally proximal on the manifold of your comprehensibility): perhaps the word "large." The transformation just performed in our minds is equivalent to that which these models attempt to simulate. Notice that removing the first word of the proposed sequence, leaving us with simply "big pizza," could instead refer to the domain of corporate pizza, demonstrating that sequential context indeed carries information. Tokenizing by word frequency simulates density, such that "big pizza" still most likely approximately means "a large pizza," just as your equation could be interpreted as pointing toward an emasculated ruler with strong empathic faculties; a concept which simply arises in written English infrequently, just as in the that which lies beneath (e.g. imagination, physical reality, and so-on).
So that's all quite a lot of words, however I fear I've left you parched for meaning; preferring to circle back around with this understanding: how do these kinds of models permit the behavior indicated by the equation in question? It's truly just as easy as aforementioned: the network features represent a transformation from the coordinate system of one manifold to another (ideally the easiest for a given dimensionality, sought, for example, with linear regression). In this case, you could loosely consider the transformation as one between a coordinate system of a sample of written language and one of (a simulacrum of) spatially contextual relative meaning. Precisely what aspects of a transformation the features represent depends, as aforementioned, largely on the technique and corpus used, and although this can vary to almost any degree one wishes it to, a wild and whacky vector space is just fine so long as we only make direct comparisons in the same vector space. Notice that a corpus's features are resultant of transformation from some other manifold (e.g. something like experiential reality spanning to written form), so by extension a simulacrum of a written language can access information about manifolds underlying itself, not exceeding the extent permitted by the transformations spanning thereto (e.g. breadth of experiences underlying the generation of the writing that constitutes the corpus). This is lovely in theory, but typically very messy in practice.
When we look at the equation you described, as in looking at most conceptual depictions of word vectors (e.g. search that in google images), it's easy to think that the vector of word "king" plus the vector of word "woman" minus the vector of the word "man" approximately equals the vector of the word "queen," but that interpretation would be severely myopic. Rather, the vector of a generalized spatially contextual relative meaning of "king" added to the same of "woman" and subtracting the same of "man" results in a vector that points toward a region of our manifold. If we try to describe what that region represents, we'll need to transform it to something we can talk about (the same kind of coordinate transformation, except done by our minds, typically called "reading"). The actual meaning of the equation becomes far more comprehensible if we pull a Baudrillard and speak in terms of a map. We can create our manifold (map) with any dimensionality, and, in the same way that latitude and longitude describe a position on a plane, we can describe our n-dimensional map with a vector for each axis. In simpler terms, think of the output of our transformation (network) as coordinates. We can do vector math like the equation in question, and the coordinates we end up with are not ambiguous. However, to talk about what's on that region, we'll need words, nearest of which---in the reference frame of written English, and for having used our corpus---is "queen." Again, we are the ones who make this transformation from our engineered manifold (machine-learnt) to one of written English (my writing this, now); we can only compare to what we know. In other words, the word2vec token nearest the coordinates of the output is "queen."
So, again, what do the coordinates on our map point to, after following the equation in question; transforming into the coordinate system of our engineered map of a spatially contextual relative understanding of written English? We could invent a word to describe precisely that point, although we apparently scarcely need one (since one does not already exist); in fact, the more precisely a word points to a meaning, the less frequently it will tend to be useful---a natural result of a quantized continuum (e.g. in choosing one number on a continuum, the probability of selecting precisely any one number goes to zero), although not exclusively influenced thereby. Again, however, if we ask which word within our corpus lies nearest to this point indicated by the coordinates produced by the equation in question, the answer (for example, using Gensim and GloVe trained on Wikipedia 2014 + Gigaword 5 (6 billion tokens and 200 dimensions) in word2vec format) is the token representing "queen," thus its approximate equality. Observe:
coordinates = pd.DataFrame()
coordinates['king'] = vectors.get_vector('king')
coordinates['woman'] = vectors.get_vector('woman')
coordinates['king+woman'] = coordinates['king'] + coordinates['woman']
coordinates['man'] = vectors.get_vector('man')
coordinates['king+woman-man'] = coordinates['king+woman'] - coordinates['man']
coordinates['queen'] = vectors.get_vector('queen')
coordinates.head() # shows the first 5 of 200 dimensions for each column
'''
+---+-----------+----------+------------+----------+----------------+-----------+
| | king | woman | king+woman | man | king+woman-man | queen |
+---+-----------+----------+------------+----------+----------------+-----------+
| 0 | -0.493460 | 0.52487 | 0.031410 | 0.10627 | -0.074860 | 0.466130 |
+---+-----------+----------+------------+----------+----------------+-----------+
| 1 | -0.147680 | -0.11941 | -0.267090 | -0.58248 | 0.315390 | -0.097647 |
+---+-----------+----------+------------+----------+----------------+-----------+
| 2 | 0.321660 | -0.20242 | 0.119240 | -0.27217 | 0.391410 | -0.072473 |
+---+-----------+----------+------------+----------+----------------+-----------+
| 3 | 0.056899 | -0.62393 | -0.567031 | -0.26772 | -0.299311 | -0.037131 |
+---+-----------+----------+------------+----------+----------------+-----------+
| 4 | 0.052572 | -0.15380 | -0.101228 | -0.11844 | 0.017212 | -0.169970 |
+---+-----------+----------+------------+----------+----------------+-----------+
'''
# it's not like the equation was referring to eigenqueen anyway...
vectors.most_similar(positive=['king', 'woman'], negative=['man'], topn=3)
'''
[('queen', 0.6978678703308105),
('princess', 0.6081745028495789),
('monarch', 0.5889754891395569)]
'''
(The similarity to 'queen' is slightly lower in the example above than in those that follow because the Gensim object's most_similar method l2-normalizes the resultant vector.)
similarity = cosine_similarity(coordinates['queen'].values.reshape((-1,200)),
coordinates['king+woman-man'].values.reshape((-1,200)))
print('Similarity: {}'.format(similarity))
# Similarity: [[0.71191657]]
# let's assign a word/token for the equation-resultant coordinates and see how it compares to 'queen'
vectors.add(entities=['king+woman-man'],weights=[coordinates['king+woman-man'].values])
distance = vectors.distance('king+woman-man','queen')
print('Distance: {}'.format(distance))
# Distance: 0.28808343410491943
# Notice that similarity and distance sum to one.
Why are the equation-resultant coordinates only 71% similar to those of the word "queen"? There are two big factors:
Firstly, by seeking to transform coordinates into a word, one attempts to make transformations inverse to those that got us to coordinates in the first place. Thus, as one can only select as correct from the discrete (tokenized) words, of which "queen" is the nearest, we settle for it. That being said, leaving our information in encoded form is fine for use in other neural networks, which adds to their practical value, and implies that word embeddings used in deep neural networks can be expected to perform slightly better in application than they do under human-language-based scrutiny.
Speaking of which, 71% isn't an especially good performance; why did it not do better? After all, is not the implication of the equation plain to see? Nonsense! The meaning we see in the equation is thoroughly embedded in our experiential understandings of reality. These models don't produce quite the results we'd like, yet better than we should've hoped for, and often entirely sufficiently for our purposes. Just as translation out of the constructed manifold into written language is cleaved as needed for translation (i.e. so we can write about where the vectors pointed, as we did just now), so, too, was meaning cleaved before our machine-learnt transformation in the first place, by nature of our having first quantized our signals in tokenization. The equation does not mean what its writer intended for it to mean. Its expressions are poorly phrased, both input and thereby output. Written as plainly as I can rightly comprehend, our translator performs marginally in this specific task (in part) because our translations both prior to and following are also marginal. We should be glad that this equation holds at all, and ought not expect as much in many intuitively logically similar cases. Observe:
vectors.most_similar(positive=['patriarch','woman'], negative=['man'], topn=31)
'''
[('orthodox', 0.5303177833557129),
('patriarchate', 0.5160591006278992),
('teoctist', 0.5025782585144043),
('maronite', 0.49181658029556274),
('constantinople', 0.47840189933776855),
('antioch', 0.47702693939208984),
('photios', 0.47631990909576416),
('alexy', 0.4707275629043579),
('ecumenical', 0.45399680733680725),
('sfeir', 0.45043060183525085),
('diodoros', 0.45020371675491333),
('bartholomew', 0.449684739112854),
('irinej', 0.4489184319972992),
('abune', 0.44788429141044617),
('catholicos', 0.4440777003765106),
('kirill', 0.44197070598602295),
('pavle', 0.44166091084480286),
('abuna', 0.4401337206363678),
('patriarchy', 0.4349902272224426),
('syriac', 0.43477362394332886),
('aleksy', 0.42258769273757935),
('melkite', 0.4203716516494751),
('patriach', 0.41939884424209595),
('coptic', 0.41715356707572937),
('abbess', 0.4165824055671692),
('archbishop', 0.41227632761001587),
('patriarchal', 0.41018980741500854),
('armenian', 0.41000163555145264),
('photius', 0.40764760971069336),
('aquileia', 0.4055507183074951),
('matriarch', 0.4031881093978882)] # <--- 31st nearest
'''
If you change 'woman' to 'female' and change 'man' to 'male', the rank falls from an already abysmal 31st to 153rd! I'll explain why in a moment. Observe that as much as we'd like to think we're dealing with relative meanings, that simply isn't correct. That doesn't mean, however, that it isn't super useful for many applications!
vectors.most_similar(positive=['metal'], negative=['genre'], topn=3)
'''
[('steel', 0.5155385136604309),
('aluminum', 0.5124942660331726),
('aluminium', 0.4897114634513855)]
'''
vectors.most_similar(positive=['metal'], negative=['material'], topn=3)
'''
[('death/doom', 0.43624603748321533),
('unblack', 0.40582263469696045),
('death/thrash', 0.3975086510181427)]
'''
# seems about right
Why such variance in performance? There isn't any; it's doing precisely what it was designed to do. The discrepancy isn't in the network, but in our expectations of it. This is the second aforementioned big factor: we see words whose meanings we know, so we think that we know the meanings of the words we see. We're returned 'queen' not because that's the word for a king who isn't a man and is a woman. Sure, there is a non-zero contribution of relative meanings, but that's a secondary action. If we aren't dealing with relative meanings, what do the outputs represent? Recall that I described the output of our transformation (network) as a "generalized spatially contextual relative meaning," the spatially contextual relativity of which is the inevitable result of the architectures and/or unsupervised mechanisms typically applied. As before, spatial relativity certainly carries some meaningful information, but written English employs many parameters in delivering meaning. If you want richer meaning to your theoretical manifolds than spatially contextual relative meaning, you'll need to design a method of supervision more suited to your desired or expected performance.
With this in mind, and looking to the code-block above, it's clear that 'metal' when referring specifically to not-'genre' produces vectors near types of metallic materials, and likewise 'metal' when referring specifically to not-'material' produces vectors near types of metal genres. This is almost entirely because tokens whose vectors are near to that of 'metal' but far from that of 'genre' seldom appear in spatial proximity with references to 'metal' as a genre, and likewise the whole lot for 'material.' In simpler terms, how often, when writing about physical metallicity, does one mention music genres? Likewise, how often, when writing about death metal (music genre) does one speak of steel or aluminum? Now it should be clear why the results of these two examples can seem so apt, while the patriarch/matriarch expectation fell flat on its face. It should also make the underlying action of the result of the equation in question quite clear.
So, all said, what is it about a model like word2vec that makes the equation hold true? Because it provides a transformation from one coordinate system to another (in this case, from a simulacrum of written English to one of spatially contextual relative meaning), which occurs frequently enough in general written English as to satisfy the given equation, behaving precisely as was intended by model architecture.
This equation is the canonical example of an application of the GloVe algorithm, and I'd argue that the algorithm does not necessarily hold.
The equation "holds" because it describes a statistical likelyhood that happens to match our held believes, which, while believes are the basis for any probabilistic expectation, on the one hand, are likely a fundamental reason for the statistical distribution of the search terms in the corpus, on the other hand.
The technical means used to arrive at this construction matter, in principle, but practicly weren't asked for, so I won't go into details, which I do anyway not know yet. Rather, you ask for a natural interpretation. I still need to describe how to arrive at that, technicly.
I would first of all assume it means that man is to king as woman is to queen--but I would expect to see quotients in word2vec (Berlin / Germany = x / France).
Instead, I interpret this to say that a king (read: kingship) without the feature a notable man (read: without the significant feature man), can be valid (only?) if it features a woman, in which case the term reduces to spheres of influence of a queen.
Although, there are likely different prototypes in the corpus (e.g. involving inferences relating woman to wife).
The more naive interpretation that queen equals king of type female where male and female are complementary is less attractive, because it also means that a king is a male queen (that's preposterous). Rather, king entails reign in terms of metonymy to a point that the network (read: I, ego, yours truly) cannot tell the difference. Of course, a king is a type of male, but the contradistinction can only be extracted from mutually exclusive instantiations. That is, more elaborate rules are necessary. Otherwise we'd only know that a king is never a woman.
