Re:ZFS Deduplication
by bensummers on Monday November 02, 2009 @10:46AM
(#916980)
The same patterns thing is mitigated by the property that changing one bit in the input is (supposed to) change on average half of the bits in the hash.
You can also be reassured that there's lots of research going on about how likely these collisions are and how to find them. People are actively trying to break these hash algorithms, so it's not just in theory.
If you hash 4KiB-long blocks, then every possible block will share the hash value with (on average) 128 different 4KiB blocks. And on a standard 200GiB disk you can fit (more or less) 52,400,000 blocks.
This explanation is a bit less reassuring. Now consider the fact that your data is never random and you hit the same patterns all the time (loads of zeros / ascii letters / x86 code)