Introduction to Locality-Sensitive Hashing

Re:Introduction to Locality-Sensitive Hashing by tylerneylon on Friday December 23, 2022 @04:44PM (#34110936)

The use case I see the most in my career is to use LSH to help solve the "ANN" problem = approximate nearest neighbors (typically with ranked results). I've seen ANN used many times for near-duplicate detection and in recommendation systems.

Although I don't have access to the proprietary code used, it's most likely that an LSH algorithm is behind the scenes in every modern search engine (to avoid serving duplicates), many modern ranking systems such as Elasticsearch (because items are typically vectorized and retrieved based on these vectors), and most recommendation systems (for similar reasons as ranking). For example, all of these pages probably have an LSH algorithm at some point (either batch processing before your request, or in some cases real-time lookups):

* Every search result page on Google * Every product page on Amazon (similar products) * All music suggestions on Spotify or similar * Every video recommendation from TikTok, YouTube, or Instagram

etc.

[ Parent | on HN ]

Re:Introduction to Locality-Sensitive Hashing by molodec on Friday December 23, 2022 @04:57PM (#34111054)

Another interesting use case for LSH is search results caching. Used by Amazon https://www.linkedin.com/feed/update/urn:li:activity:6943348... https://www.amazon.science/blog/more-efficient-caching-for-p...

[ Parent | on HN ]

Re:Introduction to Locality-Sensitive Hashing by senderista on Friday December 23, 2022 @05:47PM (#34111475)

Yes, e.g. many IR systems use cosine similarity to compute query vector/term vector similarity, and simhashing approximates cosine similarity. OTOH, some IR systems instead use a set-theoretic measure, Jacquard similarity, which can be approximated by minhashing.

[ Parent | on HN ]