ljusten
ljusten on Hacker News
Short answer, we didn't need it. While the code is largely cross-platform, there is some work involved when it gets down to the details.
We are currently working on supporting Windows to Windows. Linux to Linux has lower priority as rsync already provides all functionality, it's just a bit slower on fast connections. On slow connections, rsync and cdc_rsync perform very similarly as the sync speed is dominated by the network.
Note that cdc_rsync runs on Windows and syncs to Linux. rsync is a Linux-only tool where you'd have to jump through some hoops to make it work on Windows.
Just the compiled and baked game, not your sources. You still developed on Windows or wherever, the cloud instance was just used for running the game.
We're currently adding support for Windows to Windows to cdc_rsync. If there is demand, Linux to Linux would also be possible.
Just uncompress the binaries on the Windows machine and run cdc_rsync. The Linux component, cdc_rsync_server, is deployed automatically on first run. It is scp'ed to ~/.cache/cdc-file-transfer/bin. So nothing has to be installed on the Linux machine.
I should also note that we used a fairly fast 100 MB/sec connection to upload the data, so the rsync diffing algorithm running at 50 MB/sec is actually a bottleneck. The difference would be smaller on a slower connection, where the network overhead would dominate the results.
IIUC, rsync computes a relatively expensive Rabin-Karp rolling hash (
https://librsync.github.io/rabinkarp_8c_source.html) and performs a hash map lookup for every byte. Hash map lookups might not be very cache friendly for larger data sets. In comparison, cdc_rsync only computes
hash = (hash << 1) + random_table[data[n]];
bool chunk_boundary = (hash & magic_pattern) == 0;
per byte. That's only a few ops and very cache friendly. The random table only has 256 entries, 8 bytes each, so it easily fits into L1.
For reference, here's the paper that describes the in-place update algo in rsync:
https://www.usenix.org/legacy/events/usenix03/tech/freenix03.... I haven't looked into it more deeply, but I think it's possible to apply the same idea to variable sized chunks.
Also, most modern compression tools have an "rsyncable" option that makes the archives play more nicely with rsync.
That's because most Stadia devs used Windows, but the cloud instances ran on Linux, so devs had to copy their games from Windows to Linux. We're currently adding support for Windows to Windows as well.
Nope, luckily we still partially work from home and use Chrome Remote Desktop. The command prompt was running on my Windows machine, which I accessed remotely, so I could capture it with the extension. That wouldn't have worked if I had been in the office that day.
I got the same when I was running Stadia on a 2.4 GHz wifi network. Once I switched to a wired connection, it worked like a charm. 5 GHz supposedly works as well, but my robot vacuum cleaner needs a 2.4 GHz network facepalm.
In a nutshell, the algorithm computes
uint64_t hash = 0;
uint64_t magic_pattern = 0b001000010000100001000...;
for (size_t n = 0; n < data.size(); ++n) {
hash = (hash << 1) + random_table[data[n]];
if ((hash & magic_pattern) == 0) {
SetChunkBoundaryAt(n);
}
}
In practice, there's more bells and whistles, but that's the gist of it. By tweaking the numbers of 1's in magic_pattern you can influence the average chunk size (distance between two boundaries). With every additional 1, your chunk size halves. There's no special handling of compressed file types. You'd probably want to do that at a much higher level, e.g. just check for extensions.
>> Or, how is it that the local manages to be clairvoyant enough to choose the same windows?
Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.