Re:Similarity Search and Deduplication at Scale
by nerdponx on Saturday November 12, 2022 @08:54PM
(#33579165)
If looking for exact or near-exact duplicates, a transformer seems like it's probably overkill. Maybe it's not bad if you already have one that you can use for inference in the database, but I suspect that something as simple as Fasttext would do the job. A transformer would probably be more useful if you want to catch things like replacing words with synonyms out of a thesaurus.
Re:Similarity Search and Deduplication at Scale
by marginalia_nu on Sunday November 13, 2022 @06:47AM
(#33582166)
Wouldn't that find articles that are semantically similar, rather than structurally similar (which I interpreted GP as wanting)?
In the structural-comparison case, I imagine you might have better luck with just doing cosine similarity across the term frequency vector or some such, possibly doing random projection first to reduce dimensionality.
Or really, an LSH would do the trick.
This is one of the methods mentioned in the article. I don't have implementation experience with the other string distance measures in the article (under "normalized string" in the table), except for Q-grams. Compared to the above method Q-grams don't scale as well and are not as robust because it doesn't encapsulate an understanding of the semantics of the text.
[1] github.com/facebookresearch/faiss [2] github.com/google-research/google-research/tree/master/scann [3] www.pinecone.io