News for hackers. Stuff that matters.
Re:ZFS Deduplication by atamyrat on Monday November 02, 2009 @11:55AM (#917171)
Compress the folder with your favorite zip program and the compression ratio would be very good approximation of performance gain you will get out of zfs dedup.

Web archiving is another application that can benefit from this. You can crawl the same website 10 times a day and just store all of the files as it is.

Re:ZFS Deduplication by bensummers on Monday November 02, 2009 @12:26PM (#917278)
Compression works inside relatively short chunks of the files, compared to huge amounts of storage. Plus if you use zip, you're only looking at similarities within individual files. I don't think the results would be useful.

Incidentally, ZFS also has an option to compress the data. You have a choice of a fast but not so wonderful algorithm, or gzip level 1 to 9. Since de dupe is at the block level, I believe you can combine the two.

Re:ZFS Deduplication by atamyrat on Monday November 02, 2009 @01:20PM (#917437)
Why it wouldn't be useful?

While it might be true for some compression formats/programs, it is not the case for .tar.*, as the directory is archived to the single file first (tar), and then compressed. So if you have similarities within 2 different files, that will be exploited.

I think to make something work for "large blocks" that already does a good job for "small chunks" is just finding the right values for parameters of compression algorithm used.

Re:ZFS Deduplication by bensummers on Monday November 02, 2009 @01:23PM (#917445)
It's not a useful measure because it's not the same thing at all. Dedup only works if the block is aligned to a ZFS block size. Compression will find blocks which have no particular alignment.
Re:ZFS Deduplication by atamyrat on Monday November 02, 2009 @01:43PM (#917514)
If they were same thing, you'd get the exact result, but not an approximation.

tar.x will compress chunks of data that is smaller than zfs block size. ZFS dedup + ZFS compression will compress them as well, so what's the problem?

Re:ZFS Deduplication by patrickgzill on Monday November 02, 2009 @09:38PM (#918447)
I don't think you understand how dedupe works for cases where it is a benefit.

Imagine you have 10 VMware virtual machines, and all the VM files are stored on a ZFS server with dedupe support.

Let's say that the base install image is 2GB in size of OS, GUI, Java libraries, etc.

Non-dedupe scenario means 20GB of storage is used.

Dedupe scenario means maybe, 2.5GB of storage is used due to slight differences in the way the blocks are arranged, etc.

Now you add an 11th VM, same OS, same base setup. Storage goes up by maybe 100MB even though the VM has the same 2GB of files.

Do you see how zipping a file would not show you that effect?

Re:ZFS Deduplication by atamyrat on Tuesday November 03, 2009 @11:52AM (#919688)
note to self: Someone else compared ZFS dedup to compression on Slashdot and it scored 5/Interesting.

On HN, it is downvoted and reply with technically wrong claim gets 4 points.