Would the Optane in this case be used as an L2ARC to handle part or all of the dedupe table? I'm trying to brush up on my ZFS terminology at the moment...
Re:OpenZFS Merged to FreeBSD
by linsomniac on Tuesday August 25, 2020 @02:08PM
(#24274179)
Currently, AFAIK, the dedup table (DDT) has to reside entirely in RAM. The L2ARC is a disk-based cache of filesystem blocks (a pretty clever design, worth reading up on), but a very different thing from the DDT.
You could definitely put the L2ARC on Optane, but Optane is probably better suited to the ZIL (ZFS Intent Log, basically the journal so writes can be quickly acked) if you aren't using literal NVRAM.
Breaking the DDT out so it could exist on a Optane isn't currently available (again, AFAIK, but I recall hearing about some changes in that department, I don't recall specifics and may be misremembering).
I've basically never had a machine big enough to comfortably use dedup except on trivial loads. My backup boxes, where I'd really like to use it, have all fallen over when enabling dedup because of the RAM requirements.
ZFS 0.8 supports a feature called "Allocation Class" that makes DDT able to working on "special" storage device (aimed to use NVMe) rather than RAM. It also supports to put metadata and small blocks on special device.
https://forums.servethehome.com/index.php?threads/zfs-alloca...
My work backup server has "only" 32GB of RAM which is nowhere near enough to do dedup, even though there's a ton of stuff that could be deduplicated. Looks like it needs at least 64GB RAM to dedup, and that machine is maxed out.
But at home I'm in a similar situation to the parent. I'd like to set up a NAS, but I use it mostly for backups long term storage and don't want a big machine sitting idle all the time. A tiny Atom with Optane for cache and DDT might be ideal, in my thinking.
But for most of the storage use Backblaze is probably the right answer.