News for hackers. Stuff that matters.
Re:How does rsync work? by kzrdude on Saturday July 02, 2022 @10:05AM (#31959074)
I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?
Re:How does rsync work? by js2 on Saturday July 02, 2022 @10:55AM (#31959521)
> I guess zfs send and similar are better solutions.

It depends. I recently built a new zfs pool server and needed to transfer a few TB of data from the old pool to the new pool, but I built the new pool with a larger record size. If I’d used zfs send the files would have retained their existing record size. So rsync it was.

Re:How does rsync work? by formerly_proven on Saturday July 02, 2022 @10:17AM (#31959170)
Unless you are also doing FS-level deduplication using the same checksums, it generally makes no sense for these to be cryptographic hashes, so they're not necessarily suitable for this purpose.

IIRC neither ZFS nor btrfs use cryptographic hashes for checksumming by default.

Re:How does rsync work? by throw0101a on Saturday July 02, 2022 @10:33AM (#31959311)
> on is a short hand for fletcher4 for non-deduped datasets and sha256 for deduped datasets

* https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec...

* https://people.freebsd.org/~asomers/fletcher.pdf

* https://en.wikipedia.org/wiki/Fletcher%27s_checksum

Strangely enough SHA-512 is actually (50%) faster than -256:

> ZFS actually uses a special version of SHA512 called SHA512t256, it uses a different initial value, and truncates the results to 256 bits (that is all the room there is in the block pointer). The advantage is only that it is faster on 64 bit CPUs.

* https://twitter.com/klarainc/status/1367199461546614788

Re:How does rsync work? by slavik81 on Saturday July 02, 2022 @04:02PM (#31961862)
It seems that SHA512t256 is another name for SHA-512/256. It's a shame that the initialization is different from SHA-512, as it would have been very useful to be able to convert a SHA-512 hash into a SHA-512/256 hash.
Re:How does rsync work? by GekkePrutser on Sunday July 03, 2022 @10:49AM (#31968112)
ZFS send does not do any kind of cross checking with the receiving end so no, not really ideal. Even incremental ZFS sends don't do this, it keeps state on the sending side only. It's ok for its intended usecases but it's not a direct rsync replacement.

It's also hard/impossible to restore individual files out of a ZFS send stream without restoring the whole thing so I've reverted to using tarballs of ZFS snapshots for backups instead of ZFS send. Again, it was never really meant for this so it was my mistake trying to use it that way.