Re:NIST Retires SHA-1 Cryptographic Algorithm
by TheRealPomax on Thursday December 15, 2022 @12:37PM
(#34002915)
That changes the parameters quite a bit though. For local digests, like image deduplication of your own content, on your own computers, sha-1 is still perfectly fine. Heck, even MD5 is still workable (although more prone to collide). Nowhere in that process is the internet, or "users" or anything else like that involved =)
You use digests to quickly detect potential collisions, then you verify each collision report, then you delete the actual duplicates. Human involvement still very much required because you're curating your own data.
Re:NIST Retires SHA-1 Cryptographic Algorithm
by dylan604 on Thursday December 15, 2022 @12:59PM
(#34003214)
If we're talking specifically image deduplication, then a hash comparison is only going to find you exact matches. what about the image deduplication of trying to find alt versions of things like scaling, different codecs, etc?
if you want to dedupe images, some sort of phashing would be much better so that the actual image is considered vs just the specific bits to generate the image.
Re:NIST Retires SHA-1 Cryptographic Algorithm
by dreamcompiler on Thursday December 15, 2022 @01:14PM
(#34003407)
Very true but if the hash matches the images are guaranteed to match too. That's my first pass when deduping my drives. My second pass is looking for "logically equivalent" images.
Re:NIST Retires SHA-1 Cryptographic Algorithm
by TheRealPomax on Thursday December 15, 2022 @03:15PM
(#34005041)
Depends on the images. For photographs, a digest is enough. For "random images downloaded from the web", or when you're deduplicating lots of user's data, sure, you want data appropriate digests, like SIFT prints. But then we're back to "you had to bring network content back into this" =)
That's what the developers of subversion thought, but they didn't anticipate that once colliding files were available people would commit them to SVN repos as test cases. And then everything broke: https://www.bleepingcomputer.com/news/security/sha1-collisio...