News for hackers. Stuff that matters.
Re:Fuzzy string matching using cosine similarity by dalke on Monday September 21, 2015 @07:10AM (#10251484)
The vectors are not normalized. That is:

    from __future__ import print_function
    from collections import Counter

    def cosine_similarity_unweighted(a, b):
        a = Counter(a)
        b = Counter(b)
        dotProduct = sum((c in a) * (c in b) for c in a)
        magnitudeA = sum(1 for c in a)
        magnitudeB = sum(1 for c in b)
        return dotProduct / (magnitudeA * magnitudeB)**0.5

    def cosine_similarity_weighted(a, b):
        a = Counter(a)
        b = Counter(b)
        dotProduct = sum(a.get(c, 0) * b.get(c, 0) for c in a)
        magnitudeA = sum(value**2 for value in a.values())
        magnitudeB = sum(value**2 for value in b.values())
        return dotProduct / (magnitudeA * magnitudeB)**0.5

    print("Scores for 'hello' and 'holl'")
    print("unweighted", cosine_similarity_unweighted("hello", "holl"))
    print("weighted", cosine_similarity_weighted("hello", "holl"))
gives the output:

    Scores for 'hello' and 'holl'
    unweighted 0.866025403784
    weighted 0.925820099773
The article gives:

    holl        index: 0.9258200997725514  
which means the article used letter weights.

That said, the scores for your example are the same, because each of the letters in 'house' is used only once (the 3 in the numerator cancels out the 3 in the denominator):

    Scores for 'house' and 'hiusehiusehiuse'
    unweighted 0.8
    weighted 0.8
If the query had instead been 'houss', you would see a difference in the similarities:

    Scores for 'houss' and 'hiusehiusehiuse'
    unweighted 0.67082039325
    weighted 0.676123403783

There are no comments on this comment yet.