News for hackers. Stuff that matters.
Re:ZFS Deduplication by dcurtis on Monday November 02, 2009 @10:22AM (#916931)
Why would this be very useful? How often do you store two copies of the same data on one disk?

Edit: let me be more clear. When do you have situations where duplicate data is stored on the same disk and the best way to deal with it is through the filesystem?

If you're talking about a webapp with lots of users uploading the same photo or something, isn't that better handled before you hit the filesystem, so that you have dedupe over a number of independent disks/locations?

Re:ZFS Deduplication by zmoney on Monday November 02, 2009 @11:43AM (#917136)
It's block-level dedupe, not file-level. This is similar to how Data Domain (www.datadomain.com, recently acquired by EMC for $2.2B) does it. In their customer base, mostly medium to large storage operations, the average customer compression is more than 20x. For a small (one-disk) system your mileage will obviously vary, but even a compression factor of 3-4x saves a lot of bits.
Re:ZFS Deduplication by a2tech on Monday November 02, 2009 @10:28AM (#916943)
All the time. One of Netapp's HUGE selling points is dedupe support.
Re:ZFS Deduplication by nailer on Monday November 02, 2009 @10:41AM (#916972)
And in the SAN space too - Hitachi make a lot of money on dedupe products.
Re:ZFS Deduplication by jodrellblank on Monday November 02, 2009 @11:31AM (#917089)
One person doesn't often store two copies of the same file for no good reason.

But the great thing about this is that if operates at the block level - so if two people take a CAD drawing, change a small bit and save it to their home directories, most of the similar data can be stored once and only the changed blocks stored separately.

Re:ZFS Deduplication by bensummers on Monday November 02, 2009 @10:43AM (#916976)
Virtual machines (and Solaris Zones).

Database dumps as backups.

Email servers with lots of CCing.

Although it's probably more for 'Enterprises' than web apps.

Re:ZFS Deduplication by ponnap on Monday November 02, 2009 @11:33AM (#917096)
There are lot of use cases wherein 'dedup' helps, and it could be done in a number of ways. For eg: Lets say there is a huge email attachment and it needs to be sent to everybody in the company. Instead of saving a copy of the attachment in everybody's account, only one copy could be saved and others could point to this. Mail systems probably already do this. However, if this support natively existed in the file system layer, application developers could take advantage of this feature instead of rolling out their own 'dedup' applications.
Re:ZFS Deduplication by sp332 on Monday November 02, 2009 @11:55AM (#917175)
Software development. Multiple branches means multiple copies of a lot of files.
Re:ZFS Deduplication by wizard_2 on Monday November 02, 2009 @03:04PM (#917755)
Unless you use version control. Perforce, source safe, git, and subversion, all "dedupe" their own data. They're not all brilliant about it (try committing a copy of a large file in subversion without using `svn cp`) but zfs dedupe will be of little value on these servers. (I left out cvs because I absolutely have no idea what it does on the server.)

The real value (and the content of many EMC sales pitches) is with backing up and archiving data. EMC often argues that most "corporate" data seems not to change very much over time.

Re:ZFS Deduplication by Rexxar on Monday November 02, 2009 @03:22PM (#917794)
> Unless you use version control

I don't see how version control avoid file duplication. When you work on multiple branches at the same time. You necessarily have to retrieve each branch in different directories.

Re:ZFS Deduplication by lallysingh on Monday November 02, 2009 @08:19PM (#918342)
They don't store full copies of each revision.
Re:ZFS Deduplication by jongraehl on Wednesday November 04, 2009 @08:40PM (#923121)
I'm not sure what fool voted you down, but I fixed that. He's talking about local (checked out), not repository, copies.
Re:ZFS Deduplication by holygoat on Monday November 02, 2009 @06:21PM (#918193)
Not having to worry about it at the application level is nice.

E.g., if you store dated snapshots of content, it's much easier on you if you can just store the whole directory, rather than manually maintaining hard links back to previous versions.