News for hackers. Stuff that matters.
Re:Similarity Search and Deduplication at Scale by yeldarb on Saturday November 12, 2022 @07:55PM (#33578845)
You should be able to use embeddings for this (sort by the cosine similarity). eg OpenAI has an off the shelf offering: https://beta.openai.com/docs/guides/embeddings/what-are-embe...

We used something similar to build a “similar articles” feature & it gave us de-duplication essentially for free.

There are no comments on this comment yet.