News for hackers. Stuff that matters.
Re:Fast file synchronization and network forwarding for remo... by AnthonBerg on Sunday October 16, 2022 @03:58PM (#33226710)
I’d like to know more about the theory behind the synchronisation — how the syncing is known to be safe and non-destructive.
Re:Fast file synchronization and network forwarding for remo... by xenoscopic on Sunday October 16, 2022 @04:32PM (#33226951)
The synchronization uses a repeated three-way merge algorithm, very similar to Git's merge when merging branches. It is triggered by recursive filesystem watching, which is also used to accelerate filesystem rescans. It maintains a virtual most-recent-ancestor and uses the two synchronization endpoints as the "branches" being merged. Much like Git has "-X ours" and "-X theirs" options, Mutagen also has automated conflict resolution[0] modes that can be specified. You can find the reconciliation algorithm here[1] (and there are an exhaustive set of test cases in the corresponding _test.go file).

To avoid a large class of race conditions (at least to the extent possible allowed by POSIX and Windows), Mutagen will use `*at` style system calls for all filesystem traversal on POSIX systems, with a similar strategy on Windows.

Also, to avoid race conditions due to filesystem changes between scan time and change-application time, Mutagen will perform just-in-time checks that filesystem contents haven't changed from what was fed into the reconciliation algorithm.

[0]: https://mutagen.io/documentation/synchronization#modes [1]: https://github.com/mutagen-io/mutagen/blob/master/pkg/synchr...

Re:Fast file synchronization and network forwarding for remo... by xenoscopic on Sunday October 16, 2022 @04:37PM (#33226981)
Also, while Mutagen's exact implementation is novel in a number of ways, I would be remiss to not point out that huge amount of academic work in this field was done by Benjamin Pierce[0] and later implemented in Unison[1].

[0]: https://www.cis.upenn.edu/~bcpierce/papers/index.shtml#Synch... [1]: https://www.cis.upenn.edu/~bcpierce/unison/

Re:Fast file synchronization and network forwarding for remo... by AnthonBerg on Sunday October 16, 2022 @04:39PM (#33227004)
Thank you so much for the great replies!
Re:Fast file synchronization and network forwarding for remo... by liketochill on Sunday October 16, 2022 @06:39PM (#33227814)
I’ve been using unison for what feels like 14 years. Once working it was great but it always took me a while to figure out the exact command line options I wanted. Beautiful tool.
Re:Fast file synchronization and network forwarding for remo... by CoastalCoder on Sunday October 16, 2022 @08:26PM (#33228555)
> but it always took me a while to figure out the exact command line options I wanted.

Reminds me of my occasional use of rsync. I'm always afraid of invoking it wrong and synching in the wrong direction.