Re:Similarity Search and Deduplication at Scale
by marginalia_nu on Sunday November 13, 2022 @06:47AM
(#33582166)
Wouldn't that find articles that are semantically similar, rather than structurally similar (which I interpreted GP as wanting)?
In the structural-comparison case, I imagine you might have better luck with just doing cosine similarity across the term frequency vector or some such, possibly doing random projection first to reduce dimensionality.
In the structural-comparison case, I imagine you might have better luck with just doing cosine similarity across the term frequency vector or some such, possibly doing random projection first to reduce dimensionality.
Or really, an LSH would do the trick.