News for hackers. Stuff that matters.
Re:CDC File Transfer by geokon on Monday January 09, 2023 @12:40AM (#34307193)
This might be a dumb question.. but when I read this

"scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression."

my thinking was: If you want to send diffs.. why not just use git?

It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom diff algo? (if it's somehow not appropriate, it'd still be nice to port whatever secret sauce they use here to git..)

Re:CDC File Transfer by deathanatos on Monday January 09, 2023 @12:43AM (#34307210)
I'm sort of in the same boat, but with the sentence,

> To help this situation, we developed two tools, cdc_rsync

Why not use rsync?

(It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.)

The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happened in the rsync case/GIF, but that wasn't the case considered, so it's an oranges/apples comparison. (Or, how is it that the local manages to be clairvoyant enough to choose the same windows?)

Re:CDC File Transfer by mindwok on Monday January 09, 2023 @12:56AM (#34307280)
I was wondering the same thing. It isn't explicit in the write-up but it's because rsync does not have a native Windows implementation, only rsync under Cygwin, so they developed this to achieve the same thing except it turned out faster than rsync.
Re:CDC File Transfer by int_19h on Monday January 09, 2023 @03:55AM (#34308224)
MSYS2 has rsync. Unlike Cygwin, I would consider that native.
Re:CDC File Transfer by ljusten on Monday January 09, 2023 @03:42AM (#34308142)
>> Or, how is it that the local manages to be clairvoyant enough to choose the same windows?

Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.

Re:CDC File Transfer by mindwok on Monday January 09, 2023 @01:00AM (#34307306)
They are separate problems and not really comparable, git is a higher level tool. The diffing in git is for humans to compare code, and as such it is actually computed on the fly and not stored anywhere (unless you use git patch). Similarly, git does not provide any mechanism for sending/receiving files, it re-uses existing solutions like HTTP and SSH.

This tool exists to send/receive files, and the diffing is an implementation detail used to achieve a high level of performance. It would make more sense for git to use this library under the hood as you mentioned.

Re:CDC File Transfer by geokon on Monday January 09, 2023 @02:38AM (#34307812)
Oh right, git doesn't store diffs. So each time you push/pull, it fetches/uploads the whole file.. for every commit?

Damn..

Then I guess even using this under the hood would be essentially a rewrite or how git fundamentally works internally

Re:CDC File Transfer by pornel on Monday January 09, 2023 @08:35AM (#34310126)
There is git-pack which compresses similar files together.
Re:CDC File Transfer by rkangel on Monday January 09, 2023 @09:01AM (#34310377)
There is an analogy here, as git does deduplication 'under the hood'.

It's kind of weird actually - at the architectural level we interact with, we talk in terms of diffs - both in terms of display and what we put in.

At the next level down (content addressable store) git is storing whole files, and the git tooling translates the diffs we communicate about down into whole files for each commit.

Then at the next level down, git puts files together in packfiles (when the repo is packed) which is a compression system to make use of the fact that most files are just tweaks of other files. So, once again it's diffs.

Re:CDC File Transfer by planede on Monday January 09, 2023 @06:25AM (#34309173)
git does store deltas, but it's an implementation detail. This is what git pack files mostly are.

I don't know if git sends or receives deltas though, I think it does?

Re:CDC File Transfer by desi_ninja on Monday January 09, 2023 @01:20AM (#34307417)
Even if one tries to repurpose git for it, git doesn't work well with diffs in binaries