News for hackers. Stuff that matters.

A comment on Go Find Duplicates: A fast and simple tool to find duplicate files.

Re:Go Find Duplicates: A fast and simple tool to find duplic... by matzf on Monday August 30, 2021 @04:14PM (#28360372)
Ah yes, that's exactly what CRC32 is supposed to be used for. And it's even quicker if you don't compute it over the whole file, brilliant!
[ Parent | on HN ]
Re:Go Find Duplicates: A fast and simple tool to find duplic... by pkolaczk on Tuesday August 31, 2021 @04:41AM (#28365332)
AFAIK CRCs are not the fastest "hashes" you can get. Some non-cryptographic hashes outperform crcs by a large factor and provide longer checksums and much better statistical properties.

E.g. see this:

* https://www.strchr.com/hash_functions

* https://jpountz.github.io/lz4-java/1.2.0/xxhash-benchmark/

[ Parent | on HN ]