Edit: Here's a perhaps slightly more accessible version of the my long-winded original post:
It's just vectors, right. I can create a little gender-nobility continuum and put some words on it like so:
| gender |
| | man | woman |
| nobility +------+-------+
| | king | queen |
my_vecs = open('my_vecs.txt','w+')
my_vecs.write('4 2\nman -1.0 -1.0\nwoman 1.0 -1.0\nking -1.0 1.0\nqueen 1.0 1.0')
my_vecs = KeyedVectors.load_word2vec_format("my_vecs.txt")
results = my_vecs.most_similar(positive=['king','woman'],negative=['man'])
# ('queen', 0.9999999403953552)]
Big surprise, right? So we can skip over "how does this work," because that's easy, and get right to the deeper question with regard to mainstream practices, "how do these words get coordinates such that the equation holds?" For this, look to the training methods, which vary, but are largely spatially relational in the sequence, as in relating words in sequential proximity and otherwise. Unfortunately, this doesn't build a space of meaning like the equation hopes for, but rather builds a space where words are related (varying slightly by methods) by the frequency that a word appears in proximity to another word. That's essentially all there is to it. You can look at my code examples below to see it in action.
--- original post begins ---
As you surely recall from our previous conversations, networks produce the results you've described precisely because they were designed to, which is generally to combat all forms of ambiguity in language modeling, or, said another way, to preserve more information than can be preserved by mere tokenized word sequences. An example application objective would be to extract some information as pertains to some understanding---and I stress 'understanding' here, in that we're involving the concept of meaning from the very outset---from some sequence of text. For example, probability of spam as a function of email content, or a person's political alignment as a function of the contents of their tweets. General practices involve tokenizing words according to some criteria (e.g. order or frequency of occurrence, etc), which would be fine if words and their orders had precisely one meaning, but that's clearly a preposterous expectation of human language; not only do words have multiple (and frequently very unrelated) meanings, syntax can vary wildly and even carry meaning itself! There are many reasons why quantizing language makes for difficult comprehension and modeling. After all, it's essentially setting out to model a continuum by first quantizing all your information.
Thankfully, topological semiotics can ameliorate this. In great brevity, there are two key concepts relevant to this discussion:
- An ideal simulacrum of the physically real is as continuous as physical reality.
- "Comprehensible space" (a manifold of aggregated interpretants) receives novel input only as differentials.
The first of these, as pertains to this explanation, simply indicates (borrowing from Charles Sanders Peirce's triadic model) that an interpretant (a subjective experiential understanding of reality, if you will) should be as continuous as the object whose impressions became the signals that instigated it. Relating this to some of the aforementioned problems, consider that the meaning of the word "under" is not (in any realistically comprehensible way) related to the meanings of its constituent signs (e.g. letters), just as the meaning of "under the weather" is scarcely relatable to the meaning of its constituent signs (e.g. collocations, words, letters, and so-on); understanding the meaning of this idiom depends on knowledge of both human interaction with storms (e.g. to know that one might become ill), and an understanding of the human experience of illness (to know that this is generally undesirable). Attempting to quantize this continuous nature as a hierarchy as we tend to attempt (e.g. moments ago when I mentioned constituent signs) is both unnecessary because we can model meaning continuously, and futile because hierarchies are themselves constructs. In simpler terms: manifold learning is an ideal choice for simulating relative sign meanings.
The second above concept may seem strange and unrelated, but it carries several critical implications, of which the following is most pertinent: what is known can only exist relative to what has been known. In the more elegant words of Roland Barthes, "No sooner is a form seen than it must resemble something: humanity seems doomed to analogy." This permits imagination, but confines understanding to the space of that which has been previously experienced. In other words, experiences of reality can only exist relative to themselves; our model of language meanings can only describe meaning relative to that from which its landscape was shaped. In our application, the transformation we end up with (i.e. the features of the network), which typically receives tokenized sequences and returns vector representations within the manifold of our designing, can only provide meanings relative to the corpus on which it was trained (and, indeed, the route of navigation through that corpus), varying in depiction---which is to say, varying in the way that it describes meaning---by the method of modeling. For example, the "skipgram" model describes meaning as spatially relational context (meaning points to context), while the "continuous bag of words" model describes meaning as consisting of spatially relational context (context points to meaning).
There are obviously some heavy assumptions being made here, and not exclusively good ones. We know that relative frequency of relative sequential word position doesn't truly carry all the meanings that can be crafted into a sequence. This should come as no surprise, of course, since we're attempting to quantize a continuous relationship; creating a discrete manifold of understanding to describe continuous relationships. Shame on us, but, as you can see, it's a difficult habit to break. Nevertheless, the key take-away here is that the primary objective described above, regardless of which method you use to generate your model, is to find an equation that transforms the vector representations of tokenized sequences into vector representations of relative meanings---or, at least, the best simulacrum that a particular corpus, technique, and architecture can provide. As before, what a particular axis (or dimension) represents varies by method, and can be as arbitrary as x, y and z, or quite specific. For example, if your purposes can afford a softmax activation function, you can describe vector representations as relative constituency, and that's amusingly elegant: you could describe everything as pertains to its relationship with the words "man," "bear," and "pig," for which the mythological "man-bear-pig" might dwell somewhere in the midst. For better understanding, we can observe the same action in reverse: the secondly mentioned concept of topological semiotics indicates that an understanding of a "man-bear-pig" depends solely on understanding(s) of "man," "bear," "pig," and nothing more. As predicted, training with a softmax activation function, which is a constrained topology, indeed requires precisely that!
In terms perhaps more familiar to the linguistically inclined, consider this alternative depiction: the word "man" can produce ample interpretants, especially since the nature of interpretants should be expected to be, as aforementioned, pretty continuous. For example, the word "queen" could be used in reference to a monarch, or to a suit of playing cards, or to a person bearing such a name, among other things. Meanwhile, a queen (monarch) of the lineage "Queen" could appear more or less similar to a queen (playing card); did Lewis Carroll not evoke precisely this depiction? We can make our models high-dimensional to ameliorate the quantization inherent in dimensionality (much as how increasing the number of edges of a polygon better simulates a circle), giving more freedom for relational complexity: "man" and "woman" can reside simultaneously near to each other along some axes (e.g. such that a region might resemble "species") and distant along others (e.g. such that a region might resemble "gender"). Thankfully, we're able to understand our transforming from sign to interpretant (and so-on) because these operations are entirely self-supervised, and which is the action of understanding the meaning of what you're reading. So, then, if I ask you for a word with a meaning most closely resembling that of "big" in the phrase "a big pizza," you can consider the meaning of "big" as pertains to the given sentence, and find something very close to it (literally proximal on the manifold of your comprehensibility): perhaps the word "large." The transformation just performed in our minds is equivalent to that which these models attempt to simulate. Notice that removing the first word of the proposed sequence, leaving us with simply "big pizza," could instead refer to the domain of corporate pizza, demonstrating that sequential context indeed carries information. Tokenizing by word frequency simulates density, such that "big pizza" still most likely approximately means "a large pizza," just as your equation could be interpreted as pointing toward an emasculated ruler with strong empathic faculties; a concept which simply arises in written English infrequently, just as in the that which lies beneath (e.g. imagination, physical reality, and so-on).
So that's all quite a lot of words, however I fear I've left you parched for meaning; preferring to circle back around with this understanding: how do these kinds of models permit the behavior indicated by the equation in question? It's truly just as easy as aforementioned: the network features represent a transformation from the coordinate system of one manifold to another (ideally the easiest for a given dimensionality, sought, for example, with linear regression). In this case, you could loosely consider the transformation as one between a coordinate system of a sample of written language and one of (a simulacrum of) spatially contextual relative meaning. Precisely what aspects of a transformation the features represent depends, as aforementioned, largely on the technique and corpus used, and although this can vary to almost any degree one wishes it to, a wild and whacky vector space is just fine so long as we only make direct comparisons in the same vector space. Notice that a corpus's features are resultant of transformation from some other manifold (e.g. something like experiential reality spanning to written form), so by extension a simulacrum of a written language can access information about manifolds underlying itself, not exceeding the extent permitted by the transformations spanning thereto (e.g. breadth of experiences underlying the generation of the writing that constitutes the corpus). This is lovely in theory, but typically very messy in practice.
When we look at the equation you described, as in looking at most conceptual depictions of word vectors (e.g. search that in google images), it's easy to think that the vector of word "king" plus the vector of word "woman" minus the vector of the word "man" approximately equals the vector of the word "queen," but that interpretation would be severely myopic. Rather, the vector of a generalized spatially contextual relative meaning of "king" added to the same of "woman" and subtracting the same of "man" results in a vector that points toward a region of our manifold. If we try to describe what that region represents, we'll need to transform it to something we can talk about (the same kind of coordinate transformation, except done by our minds, typically called "reading"). The actual meaning of the equation becomes far more comprehensible if we pull a Baudrillard and speak in terms of a map. We can create our manifold (map) with any dimensionality, and, in the same way that latitude and longitude describe a position on a plane, we can describe our n-dimensional map with a vector for each axis. In simpler terms, think of the output of our transformation (network) as coordinates. We can do vector math like the equation in question, and the coordinates we end up with are not ambiguous. However, to talk about what's on that region, we'll need words, nearest of which---in the reference frame of written English, and for having used our corpus---is "queen." Again, we are the ones who make this transformation from our engineered manifold (machine-learnt) to one of written English (my writing this, now); we can only compare to what we know. In other words, the word2vec token nearest the coordinates of the output is "queen."
So, again, what do the coordinates on our map point to, after following the equation in question; transforming into the coordinate system of our engineered map of a spatially contextual relative understanding of written English? We could invent a word to describe precisely that point, although we apparently scarcely need one (since one does not already exist); in fact, the more precisely a word points to a meaning, the less frequently it will tend to be useful---a natural result of a quantized continuum (e.g. in choosing one number on a continuum, the probability of selecting precisely any one number goes to zero), although not exclusively influenced thereby. Again, however, if we ask which word within our corpus lies nearest to this point indicated by the coordinates produced by the equation in question, the answer (for example, using Gensim and GloVe trained on Wikipedia 2014 + Gigaword 5 (6 billion tokens and 200 dimensions) in word2vec format) is the token representing "queen," thus its approximate equality. Observe:
coordinates = pd.DataFrame()
coordinates['king'] = vectors.get_vector('king')
coordinates['woman'] = vectors.get_vector('woman')
coordinates['king+woman'] = coordinates['king'] + coordinates['woman']
coordinates['man'] = vectors.get_vector('man')
coordinates['king+woman-man'] = coordinates['king+woman'] - coordinates['man']
coordinates['queen'] = vectors.get_vector('queen')
coordinates.head() # shows the first 5 of 200 dimensions for each column
| | king | woman | king+woman | man | king+woman-man | queen |
| 0 | -0.493460 | 0.52487 | 0.031410 | 0.10627 | -0.074860 | 0.466130 |
| 1 | -0.147680 | -0.11941 | -0.267090 | -0.58248 | 0.315390 | -0.097647 |
| 2 | 0.321660 | -0.20242 | 0.119240 | -0.27217 | 0.391410 | -0.072473 |
| 3 | 0.056899 | -0.62393 | -0.567031 | -0.26772 | -0.299311 | -0.037131 |
| 4 | 0.052572 | -0.15380 | -0.101228 | -0.11844 | 0.017212 | -0.169970 |
# it's not like the equation was referring to eigenqueen anyway...
vectors.most_similar(positive=['king', 'woman'], negative=['man'], topn=3)
(The similarity to 'queen' is slightly lower in the example above than in those that follow because the Gensim object's
most_similar method l2-normalizes the resultant vector.)
similarity = cosine_similarity(coordinates['queen'].values.reshape((-1,200)),
# Similarity: [[0.71191657]]
# let's assign a word/token for the equation-resultant coordinates and see how it compares to 'queen'
distance = vectors.distance('king+woman-man','queen')
# Distance: 0.28808343410491943
# Notice that similarity and distance sum to one.
Why are the equation-resultant coordinates only 71% similar to those of the word "queen"? There are two big factors:
Firstly, by seeking to transform coordinates into a word, one attempts to make transformations inverse to those that got us to coordinates in the first place. Thus, as one can only select as correct from the discrete (tokenized) words, of which "queen" is the nearest, we settle for it. That being said, leaving our information in encoded form is fine for use in other neural networks, which adds to their practical value, and implies that word embeddings used in deep neural networks can be expected to perform slightly better in application than they do under human-language-based scrutiny.
Speaking of which, 71% isn't an especially good performance; why did it not do better? After all, is not the implication of the equation plain to see? Nonsense! The meaning we see in the equation is thoroughly embedded in our experiential understandings of reality. These models don't produce quite the results we'd like, yet better than we should've hoped for, and often entirely sufficiently for our purposes. Just as translation out of the constructed manifold into written language is cleaved as needed for translation (i.e. so we can write about where the vectors pointed, as we did just now), so, too, was meaning cleaved before our machine-learnt transformation in the first place, by nature of our having first quantized our signals in tokenization. The equation does not mean what its writer intended for it to mean. Its expressions are poorly phrased, both input and thereby output. Written as plainly as I can rightly comprehend, our translator performs marginally in this specific task (in part) because our translations both prior to and following are also marginal. We should be glad that this equation holds at all, and ought not expect as much in many intuitively logically similar cases. Observe:
vectors.most_similar(positive=['patriarch','woman'], negative=['man'], topn=31)
('matriarch', 0.4031881093978882)] # <--- 31st nearest
If you change 'woman' to 'female' and change 'man' to 'male', the rank falls from an already abysmal 31st to 153rd! I'll explain why in a moment. Observe that as much as we'd like to think we're dealing with relative meanings, that simply isn't correct. That doesn't mean, however, that it isn't super useful for many applications!
vectors.most_similar(positive=['metal'], negative=['genre'], topn=3)
vectors.most_similar(positive=['metal'], negative=['material'], topn=3)
# seems about right
Why such variance in performance? There isn't any; it's doing precisely what it was designed to do. The discrepancy isn't in the network, but in our expectations of it. This is the second aforementioned big factor: we see words whose meanings we know, so we think that we know the meanings of the words we see. We're returned 'queen' not because that's the word for a king who isn't a man and is a woman. Sure, there is a non-zero contribution of relative meanings, but that's a secondary action. If we aren't dealing with relative meanings, what do the outputs represent? Recall that I described the output of our transformation (network) as a "generalized spatially contextual relative meaning," the spatially contextual relativity of which is the inevitable result of the architectures and/or unsupervised mechanisms typically applied. As before, spatial relativity certainly carries some meaningful information, but written English employs many parameters in delivering meaning. If you want richer meaning to your theoretical manifolds than spatially contextual relative meaning, you'll need to design a method of supervision more suited to your desired or expected performance.
With this in mind, and looking to the code-block above, it's clear that 'metal' when referring specifically to not-'genre' produces vectors near types of metallic materials, and likewise 'metal' when referring specifically to not-'material' produces vectors near types of metal genres. This is almost entirely because tokens whose vectors are near to that of 'metal' but far from that of 'genre' seldom appear in spatial proximity with references to 'metal' as a genre, and likewise the whole lot for 'material.' In simpler terms, how often, when writing about physical metallicity, does one mention music genres? Likewise, how often, when writing about death metal (music genre) does one speak of steel or aluminum? Now it should be clear why the results of these two examples can seem so apt, while the patriarch/matriarch expectation fell flat on its face. It should also make the underlying action of the result of the equation in question quite clear.
So, all said, what is it about a model like word2vec that makes the equation hold true? Because it provides a transformation from one coordinate system to another (in this case, from a simulacrum of written English to one of spatially contextual relative meaning), which occurs frequently enough in general written English as to satisfy the given equation, behaving precisely as was intended by model architecture.