News for hackers. Stuff that matters.
Re:Bup – towards the perfect backup by fragmede on Monday November 17, 2014 @05:47PM (#8621226)
> That is a dataset which is already deduplicated via copy-on-write semantics (it was not using ZFS deduplication because you should basically never use ZFS deduplication).

Can someone more experienced with ZFS say why?

Re:Bup – towards the perfect backup by aidenn0 on Monday November 17, 2014 @06:24PM (#8621424)
"Basically never" is an overstatement, but it is true to the point of "Never unless you already know why I said 'basically never'"

It boils down to the fact that ZFS maintains a mapping from hashes to LBNs. This allows write-time deduplication (as opposed to a scrubber that runs periodically and retroactively deduplicates already written blocks). This is somewhat memory intensive though. For smaller ZFS pools you can get away with just having lots of RAM (and with or without dedupe ZFS performs better the more RAM you have). For larger ones, you can add a SSD to act as additional disk cache.

Here's a quick description of that setup:

https://blogs.oracle.com/brendan/entry/test

Note in this example that they were already showing 128GB of RAM for a 17TB pool; the L2ARC was to augment that. In general, ZFS was designed with a much higher RAM/Disk ratio than a workstation typically has.

Re:Bup – towards the perfect backup by pwildani on Monday November 17, 2014 @07:03PM (#8621635)
ZFS is also very far away from the state of the art in online dedup. For instance, http://users.soe.ucsc.edu/~avani/wildani-icde13dedup.pdf has a theoretical dedup regime that needs only 1% of the RAM for 90% of the benefit.
Re:Bup – towards the perfect backup by mgbmtl on Monday November 17, 2014 @05:58PM (#8621295)
I'm not an expert by any means, but the most cited reasons are that it requires a very big amount of ram, and it depends a lot (obviously) on the type of data.
Re:Bup – towards the perfect backup by timewasted on Monday November 17, 2014 @06:25PM (#8621430)
In addition to the memory requirements, I seem to recall that it works at the block level, as opposed to the file level. So you could have two of the same file, but maybe one copy is written at the start of a block and one is written in the middle of a block. Same file, different blocks, so no deduplication.
Re:Bup – towards the perfect backup by aidenn0 on Monday November 17, 2014 @08:09PM (#8621876)
I'm pretty sure that ZFS doesn't do block-suballocation, so that scenario is not possible.