News for hackers. Stuff that matters.
Re:Show HN: All-pair similarity search on millions of sets i... by ekzhu on Sunday November 18, 2018 @11:42PM (#18484268)
Author here. The algorithm used here is based on Google's 2007 paper "Scaling Up All Pairs Similarity Search." Since then I am sure they have started to look at billions of sets. Generally speaking exact algorithms like the one presented here max-out around 100M on not-crazy hardware, but going over a billion probably requires some approximate algorithms such as Locality Sensitive Hashing. You maybe interested in the work by Anshumali Shrivastava.

There are no comments on this comment yet.