Re:An Introduction to ZFS
by benlivengood on Friday November 20, 2020 @11:59AM
(#25162314)
> In addition, they don't actually bring that much space saving on common workloads. Most people don't have a lot of truly identical blocks of data.
VM backing storage seems to be the biggest worthwhile use case, but that depends on whether snapshots and clones are used extensively. Installing 100 copies of Debian on empty VMs will likely get deduped quite a bit. But it's faster and provides almost the same benefits to install one VM, snapshot it, and produce the rest of the VMs by cloning from the snapshot.
The only other case I could imagine dedup being good for is storing a lot of genomic data: https://techtransfer.universityofcalifornia.edu/NCD/25080.ht...
But if the use-case is narrow enough for custom deduplication then it will probably be much more efficient than ZFS's block-based dedup.
No expert either but from what I've gathered the main issue is that the dedup tables require a _lot_ of memory that scales with the size of the pool, and if they can't fit in RAM performance tanks. However the real issue is that any blocks written while dedup was enabled will demand this dedup overhead, even if the dedup option is turned off. That is, it's a "sticky feature".
So if you use dedup, notice performance tanks because dedup tables are too big, well you're screwed because turning it off won't give you back the performance. Only way to recover is to send/receive the whole shebang to a separate pool.
In addition, they don't actually bring that much space saving on common workloads. Most people don't have a lot of truly identical blocks of data.