Re:Show HN: All-pair similarity search on millions of sets i...
by ekzhu on Sunday November 18, 2018 @11:42PM
(#18484268)
Author here. The algorithm used here is based on Google's 2007 paper "Scaling Up All Pairs Similarity Search." Since then I am sure they have started to look at billions of sets. Generally speaking exact algorithms like the one presented here max-out around 100M on not-crazy hardware, but going over a billion probably requires some approximate algorithms such as Locality Sensitive Hashing. You maybe interested in the work by Anshumali Shrivastava.
[ Parent
| on HN ]
There are no comments on this comment yet.