Re:Ask HN: Best tool for de-duplication of local files?
by Someone on Thursday November 03, 2022 @01:24PM
(#33454337)
I would first remove exact duplicates by generating a database of hashes for every single file, and automatically discard duplicates.
Once you have removed the easy duplicates, use perceptual hashing (https://en.wikipedia.org/wiki/Perceptual_hashing) to find likely copies. Then, either inspect duplicates manually to check whether they are indeed variants of the same, or just assume they are. When you have the set of (assumed) duplicates, either manually pick the best one or just keep the largest file (that works reasonably well for .jpg files, but may discard better images if, say, the original JPEG 2000 file was converted to a smaller .png)
I would also keep all original paths, format, size for every disambiguated file. That may come in handy later, and shouldn’t add much.
Once you have removed the easy duplicates, use perceptual hashing (https://en.wikipedia.org/wiki/Perceptual_hashing) to find likely copies. Then, either inspect duplicates manually to check whether they are indeed variants of the same, or just assume they are. When you have the set of (assumed) duplicates, either manually pick the best one or just keep the largest file (that works reasonably well for .jpg files, but may discard better images if, say, the original JPEG 2000 file was converted to a smaller .png)
I would also keep all original paths, format, size for every disambiguated file. That may come in handy later, and shouldn’t add much.