Re:Fuzzy string matching using cosine similarity
by dalke on Monday September 21, 2015 @07:10AM
(#10251484)
The vectors are not normalized. That is:
from __future__ import print_function
from collections import Counter
def cosine_similarity_unweighted(a, b):
a = Counter(a)
b = Counter(b)
dotProduct = sum((c in a) * (c in b) for c in a)
magnitudeA = sum(1 for c in a)
magnitudeB = sum(1 for c in b)
return dotProduct / (magnitudeA * magnitudeB)**0.5
def cosine_similarity_weighted(a, b):
a = Counter(a)
b = Counter(b)
dotProduct = sum(a.get(c, 0) * b.get(c, 0) for c in a)
magnitudeA = sum(value**2 for value in a.values())
magnitudeB = sum(value**2 for value in b.values())
return dotProduct / (magnitudeA * magnitudeB)**0.5
print("Scores for 'hello' and 'holl'")
print("unweighted", cosine_similarity_unweighted("hello", "holl"))
print("weighted", cosine_similarity_weighted("hello", "holl"))
gives the output:
Scores for 'hello' and 'holl'
unweighted 0.866025403784
weighted 0.925820099773
The article gives:
holl index: 0.9258200997725514
which means the article used letter weights.
That said, the scores for your example are the same, because each of the letters in 'house' is used only once (the 3 in the numerator cancels out the 3 in the denominator):
Scores for 'house' and 'hiusehiusehiuse'
unweighted 0.8
weighted 0.8
If the query had instead been 'houss', you would see a difference in the similarities:
Scores for 'houss' and 'hiusehiusehiuse'
unweighted 0.67082039325
weighted 0.676123403783
That said, the scores for your example are the same, because each of the letters in 'house' is used only once (the 3 in the numerator cancels out the 3 in the denominator):
If the query had instead been 'houss', you would see a difference in the similarities: