Re:Finding unique items: hash vs. sort
by jlokier on Wednesday January 08, 2020 @07:46AM
(#21990879)
> But calculating a SHA256 takes time linear in the length of the thing you're hashing.
Ahem, so does hashing.
And so does sort comparison.
The GP's idea is actually quite effective for medium-to-large objects.
As a bonus, you can cache and store the SHA256 of each object for future in-set testing and uniqueness-finding without looking at the objects themselves. (You cannot do this with non-cryptographic hashes or sorting by comparisons.)
Ahem, so does hashing.
And so does sort comparison.
The GP's idea is actually quite effective for medium-to-large objects.
As a bonus, you can cache and store the SHA256 of each object for future in-set testing and uniqueness-finding without looking at the objects themselves. (You cannot do this with non-cryptographic hashes or sorting by comparisons.)
This is basically what Git does.