Re:Similarity Search and Deduplication at Scale
by nerdponx on Saturday November 12, 2022 @08:54PM
(#33579165)
If looking for exact or near-exact duplicates, a transformer seems like it's probably overkill. Maybe it's not bad if you already have one that you can use for inference in the database, but I suspect that something as simple as Fasttext would do the job. A transformer would probably be more useful if you want to catch things like replacing words with synonyms out of a thesaurus.
[ Parent
| on HN ]
There are no comments on this comment yet.