What do WordNet::Similarity scores mean?

I am using WordNet Interface in NLTK, which facilitates computation of a number of similarity metrics:

• Path similarity
• Leacock-Chodorow Similarity
• Wu-Palmer Similarity
• Resnik Similarity
• Jiang-Conrath Similarity
• Lin Similarity

I tried computing all of them on words w1 = 'love', w2 = 'hate', w3 = 'romance'. On all the metrics, the similarity scores obtained were higher for the pair 'love' and 'hate' rather than the 'love' and 'romance'. Shouldn't it have been the opposite case?

The most accessible resource that explains the difference between each of these word similarity metrics would be Dan Jurafsky and James H. Martin's ubiquitous Speech and Language Processing 2nd Edition. Specifically, pages 652-667 in chapter 20 (Computational Lexical Semantics) briefly and comprehensively cover each metric/algorithm in a way that anyone with just a basic understanding of math, language, and graphs can understand.

I will do my best to summarize each metric, using Jurafsky and Martin as my primary citation (attribute all my summaries to that book), annotated with my own understanding / insight where relevant / useful.

Broadly we can group the metrics based on what parameters they operate on. Roughly there are two groups: (1) metrics which use only a thesaurus (e.g. WordNet) and (2) which use a thesaurus and probabilistic information from distributions in corpora.

These metrics belong to the thesaurus-based ones:

• Path similarity
• Leacock-Chodorow Similarity (Leacock and Chodorow 1998)
• Wu-Palmer Similarity (Wu and Palmer 1994)

How do these work?

Path similarity computes shortest number of edges from one word sense to another word sense, assuming a hierarchical structure like WordNet (essentially a graph). In general, word senses which have a longer path distance are less similar than those with a very short path distance, e.g. man, dog versus man, tree (expectation is that man is more similar to dog than it is to tree).

The path similarity can be defined as:

sim$_{\text{path}}(c_1, c_2) = \text{pathlen}(c_1, c_2)$

where $c_1$, $c_2$ are word senses, and $\text{pathlen}(c_1, c_2)$ is the shortest number of edges between those two word senses in a given thesaurus like WordNet.

Leacock-Chodorow Similarity, or LCH, is practically the same thing, except it uses the negative logarithm of the result of path similarity.

sim$_{\text{path}}(c_1, c_2) = -log \text{pathlen}(c_1, c_2)$

The negative logarithm is in the domain of information theory.

The Wu-Palmer metric (WUP) is very similar to LCH, except it weights the edges based on distance in the hierarchy. For example, jumping from inanimate to animate is a larger distance than jumping from say Felid to Canid. In some sense we can think of it as sort of edit distance, assigning type changing operations a higher cost the higher they are in the hierarchy.

These metrics belong to the thesaurus- and corpus-based ones (also called the Information Content metrics):

• Resnik Similarity (Resnik 1995)
• Lin Similarity (Lin 1998b)
• Jiang-Conrath distance (Jiang and Conrath 1997)

Basically each of these algorithms center around calculating the probability of the lowest common subsumer between two word senses $c_1$ and $c_2$, which is the lowest node in the hierarchy that is the parent of both $c_1$ and $c_2$. The probability comes from the distribution sampled from, i.e. the corpora used. Resnik is the simplest implementation of this, while Lin expands it by considering similarity as both the information content shared between two senses, and the difference. Jiang-Conrath is actually a distance function, best summarized from the chapter:

Jiang-Conrath distance (Jiang and Conrath, 1997), although derived in a completely different way from Lin and expressed as a distance rather than a similarity function, has been shown to work well or better than all the other thesaurus-based methods (p. 656)

I tried computing all of them on words w1 = 'love', w2 = 'hate', w3 = 'romance'. On all the metrics, the similarity scores obtained were higher for the pair 'love' and 'hate' rather than the 'love' and 'romance'. Shouldn't it have been the opposite case?

Jurafsky and Martin answer this question directly in the chapter:

Word relatedness characterizes a larger set of potential relationships between words: antonyms...have a higher relatedness but low similarity...Word similarity is thus a sub case of word relatedness. In general, the five algorithms we describe in this section do not attempt to distinguish between similarity and semantic relatedness. (p. 653)

Using this (and the basic understanding of each similarity metric), the word senses of love and hate, while antonyms, are very related, since they essentially belong to the same semantic type (one's feeling for something else). Thus, it would be expected that the metrics give a higher similarity to them, than say love and romance; romance is very similar to love, but its type is not as close as say hate or dislike. This hypothesis is quickly confirmed by testing the WUP metric:

>>> wn.wup_similarity(dislike, love)
0.7692307692307693
>>> wn.wup_similarity(romance, love)
0.25


I would leave it as an exercise to compute this for other metrics, but I am 100% certain you will get similar results for any other metric. What you could do is investigate metrics that consider similarity and not relatedness, i.e. would rank the similarity of love, romance higher than love, hate. Jurafsky and Martin don't seem to give any references to papers about this, however.

Hope this helps.