Do you use Restic or Borg with ZFS? How do you have that set up? Do you use them in lieu of zfs send/recv?

In liu of ZFS replication (send and receive). I still use ZFS as a file system.

Also, I have a 1.5TB directory. There is a lot of redundant backup data in it. I archived it with Restic to 400 GB. As a ZFS dataset, it would have taken ~4X the size.

Honestly, backups up to 100 TB are better be done with tools such as Restic than file system stream backups: less hardware requirements, lots of support for repository management, integration with clouds, portable repositories, better and more trusted encryption, etc. Tools based on Go are static binaries with no dependencies. You can recover your data in the future on any X86 platform.

Fully agree on using tools such as Restic! The `--exclude-caches` is really helpful in keeping backups small, it makes Restic skip directories with a CACHEDIR.TAG file and that includes Rust compile target directories. Combined with a small exclude list for browser caches and other temporary storage this makes backup deltas way smaller than in the average ZFS setup. (And no, creating a new ZFS filesystem for every cache directory and excluding each of them from snapshots is not really a solution)

I searched my linux system for CACHEDIR.TAG and found only a handful of them. Also looking at the standard’s website, it seems it’s not adopted.

I make remote snapshot backups with Borg using this: https://github.com/Freaky/zfsnapr

zfsnapr mounts recursive snapshots on a target directory so you can just point whatever backup tool you like at a normal directory tree.

I still use send/recv for local backups - I think it's good to have a mix of strategies.

I’m not sure what restic or borg are or do for you, but there is an upcoming feature called “block cloning” that is a sort of on-demand version deduplication that offers new advantages for certain setups.

https://www.bsdcan.org/events/bsdcan_2023/sessions/session/1...

Borg/Restic are deduplicating backup tools. They actually use the exact same approach as ZFS deduplication, but because e.g. Borg chunks are around 2M instead of 128K (ZFS recordsize default) it uses less memory for the same data compared to ZFS. All of these systems end up using somewhere around 300 bytes of memory per chunk stored. This is how you arrive at the "1 GB of memory per 1 TB of storage" rule of thumb when using deduplication.

Block cloning is different because it's offline. It does not require deduplication tables, therefore does not require those 300 bytes per chunk I mentioned above, neither on-disk nor in-core. Note that the BRT only has to track blocks which are actually referenced at least twice, while the DDT has to track all blocks. This, plus the BRT not having to store the block checksum, is why it has so much less overhead. The BRT should come out to less than 20 bytes per tracked block on average. (Also, the BRT is only used for freeing blocks and when handling ficlonerange, while the DDT has to be consulted for every prospective block write).

The downside is of course that it's offline. So the finding of duplicate blocks, which is what all the DDT overhead supports, has to be done by an external tool, unless you're explicitly creating duplicates (cp --reflink=always), which is of course a very useful thing to have -- especially because block cloning works across datasets: with block cloning you can actually copy data out of snapshots back into a writable dataset without having to physically copy anything. Also, again, as it's offline, you can deduplicate existing data in-place, while traditional ZFS dedup would require turning deduplication on and then re-writing all data you want deduplicated.

So OpenZFS 2.2 is definitely going to be very exciting.