Re:Similarity Search and Deduplication at Scale
by nerdponx on Saturday November 12, 2022 @08:52PM
(#33579155)
Exact and near duplicate articles should have similar or identical word frequency distributions. Maybe that can be used as a blocking criterion somehow. Although it might not be any faster to compare word frequency distributions than to compare dense low-dimensional embeddings.
[ Parent
| on HN ]
There are no comments on this comment yet.