News for hackers. Stuff that matters.
Re:Similarity Search and Deduplication at Scale by nerdponx on Saturday November 12, 2022 @08:54PM (#33579165)
If looking for exact or near-exact duplicates, a transformer seems like it's probably overkill. Maybe it's not bad if you already have one that you can use for inference in the database, but I suspect that something as simple as Fasttext would do the job. A transformer would probably be more useful if you want to catch things like replacing words with synonyms out of a thesaurus.

There are no comments on this comment yet.