News for hackers. Stuff that matters.
Re:ZFS Deduplication by bensummers on Monday November 02, 2009 @10:42AM (#916973)
Happily, they have a 'verify' mode which checks the data on disc instead of making the assumption that SHA256 works as promised.
Re:ZFS Deduplication by m_eiman on Monday November 02, 2009 @11:37AM (#917114)
They do, but what I'm interested in is if it's something that actually occurs in real data (with SHA256, in the article he says that it's of more use with "worse but faster" hashing algorithms).

My initial remedy is to add another hash method and name the data based on the results of both; that way problematic data would need to trigger a collision in two different algorithms at the same time, which "should" be next to impossible. Currently my file fragments are named "{SHA256}.dat", in v2 I could instead name them "{SHA256}{SOME_OTHER_HASH}.dat".

Re:ZFS Deduplication by jcsiracusa on Monday November 02, 2009 @12:35PM (#917305)
Regarding the "should" part: http://stackoverflow.com/questions/1323013
Re:ZFS Deduplication by m_eiman on Monday November 02, 2009 @01:17PM (#917430)
Good stuff! Perhaps it's safe to remove the "" qualifier from my assumption.