I was trying to complete an NLP assignment using the Jaccard Distance metric function
jaccard_distance() built into
nltk.metrics.distance, when I noticed that the results from it did not make sense in the context I would expect.
Specifically, the implementation in
return (len(label1.union(label2)) - len(label1.intersection(label2)))/len(label1.union(label2))
but according to the definition, the numerator term should only involve an intersection of the two sets, which means the correct implementation should be:
when I wrote my own function using the latter, I indeed obtained correct answers to my assignment. For example, I was tasked to recommend a correct spelling suggestion for the misspelled word cormulent, from a comprehensive corpus of words (built in
nltk), using Jaccard Distance on trigrams of the words.
When I used the
nltk, I instead obtained so many perfect matches (the result from the distance function was
1.0) that just were nowhere near being correct.
When I used my own function the latter implementation, I was able to get a spelling recommendation of corpulent, at a Jaccard Distance of 0.4 from cormulent, a decent recommendation.
Could there be a bug with