News for hackers. Stuff that matters.

A comment on Software Deduplication: Quick comparison of save ratings. Parent comment

Re:Software Deduplication: Quick comparison of save ratings by vardump on Wednesday February 10, 2016 @07:13PM (#11077295)
The files on CDs are aligned to 2 kB boundaries. Dedup is looking for n kB continuous block. If the block size of the material you want to dedup does not match to the block size of dedup system, you'll get suboptimal results. The bigger the difference, the worse the results.

Say you have this data:

  ABCABCBACCBBABCC
Dedup system that has block size of 1 can see you really have just three unique blocks, A, B and C.

Same data, but dedup with block size of 2:

  AB CA BC BA CC BB AB CC
Dedup block size of 2 thinks you have 6 unique blocks: AB, CA, BC, BA, CC and BB.

Etc.

[ Parent | on HN ]
Re:Software Deduplication: Quick comparison of save ratings by linuxready on Wednesday February 10, 2016 @07:25PM (#11077356)
I'm sorry I am not sure I get it. Let's say you have a 1000 kB file which is duplicated and which is located on continuous blocks (so if the CDs used 2 kB boundaries, we'll have 500 continuous blocks). If ZFS use 128 kB block size, it will detect 7 blocks (896 kB) that it can deduplicate. So we only lose about 10%.

Perhaps there is a high degree of fragmentation then and files are not on continous blocks ?

(this example would be the same if instead of 2 exactly duplicated files, we have a big common chunk between 2 files)

[ Parent | on HN ]
Re:Software Deduplication: Quick comparison of save ratings by vardump on Wednesday February 10, 2016 @07:28PM (#11077366)
Wrong. If the alignment is wrong, you'll likely lose 100%. 2 kB can be aligned in 64 different positions within 128 kB.

That 1000 kB of 2 kB continuous blocks must start exactly at same mod 128 kB alignment. There are 64 different possible alignments.

[ Parent | on HN ]
Re:Software Deduplication: Quick comparison of save ratings by linuxready on Wednesday February 10, 2016 @07:31PM (#11077375)
Oh that's it then ! Thanks for the clarification.
[ Parent | on HN ]