Unison File Synchronizer

Re:Unison File Synchronizer by sedachv on Tuesday August 06, 2019 @03:47PM (#20628794)

> - Unison synchronizes files to files, but for a Dropbox-like system you really want deduplication for space savings (i.e. server-side storage is a bunch of pointers to content-addressed blocks.) > - in general, it's not clear to me that the client is really the hard part of Dropbox. Note e.g. the part where Dropbox now runs its own data centers for cost reasons.

The whole point of Unison is that you do not need a server. And certainly not a server run by a for-profit corporation in the United States of Surveillance.

[ Parent | on HN ]

Re:Unison File Synchronizer by mindslight on Tuesday August 06, 2019 @04:57PM (#20629553)

Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions.

I agree it doesn't have to be a corporate-owned, or even corporate-snoopable "cloud" server, and if this is your goal then unison will work well. Also you can trivially solve the dedup using zfs, but I don't think dedup is a killer feature for personal storage.

(Better would be a "manual dedup" utility to catch those instances where you copied off the same thing multiple times over the years. I'd appreciate any recommendations here, although writing one seems trivial when I get around to it - calculate recursive sha256sums for every node in the filesystem tree, and look for the biggest matching ones)

[ Parent | on HN ]

Re:Unison File Synchronizer by sedachv on Thursday August 08, 2019 @09:53PM (#20650696)

> Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions.

You do not need a spanning tree or loops, you just need to do a topological sort on the nodes/disks and then consistently sync them in that order. There is no magic algorithm that will handle conflict resolution for arbitrary syncing.

[ Parent | on HN ]

Re:Unison File Synchronizer by mindslight on Friday August 09, 2019 @11:52AM (#20655482)

The problem I'm referring to isn't conflict resolution, but rather spurious re-creation of files that have been deleted. Unison's pairwise syncing cannot distinguish a file that has been deleted from one that has yet to be created (whereas say vector clocks do).

Create file F on A. Sync A-B, A-C. Delete F on A. Sync A-B. ?????. Sync B-C. Sync A-B. File F now re-exists on A (and B and C).

??? is some event where you cannot sync A-C to directly save A's changes on C, yet you still want to save any changes from B on C. Say a remote machine crashes and becomes unavailable, you didn't have the time over 3G, or some other type of ad-hoc craziness which is why you're using distributed syncing in the first place.

Which implies you need to choose one node from {A,B,C} that is the most likely to be available to sync the other two to. That node can also provide connectivity to a larger network, which generalizes to a spanning tree.

IMO a topological sort would be an even stricter requirement than spanning tree, in that if one node becomes available, you can't sync anything "below" it! Also what does syncing disks "in order" mean when changes can happen at any time? (eg I use unison for maildir).

[ Parent | on HN ]

Re:Unison File Synchronizer by sedachv on Saturday August 10, 2019 @04:27PM (#20664350)

> The problem I'm referring to isn't conflict resolution, but rather spurious re-creation of files that have been deleted.

That is conflict resolution, over the set of files. Which is why Unison provides the "reconciling changes" UI.

> Which implies you need to choose one node from {A,B,C} that is the most likely to be available to sync the other two to.

Yes, that is a way to avoid update conflicts.

> IMO a topological sort would be an even stricter requirement than spanning tree, in that if one node becomes available, you can't sync anything "below" it!

Yes, it is also a way to avoid update conflicts.

> (eg I use unison for maildir).

Why? IMAP already takes care of synchronization, without any potential problems.

[ Parent | on HN ]

Re:Unison File Synchronizer by mindslight on Thursday August 15, 2019 @05:58PM (#20710116)

Eh, we can disagree on a precise definition of abstract "conflict resolution", but I'm referring to unison's behavior with -batch. There's no logical conflict that needs to be resolved by user choice or algorithm. When looking at the larger system of all the replicas, unison's behavior is to simply undelete a file that was explicitly deleted.

> Why? IMAP already takes care of synchronization, without any potential problems.

Less configuration, less attack surface. Sharing mail over NFS is a common thing, and I view unison as a type of distributed filesystem.

[ Parent | on HN ]

Re:Unison File Synchronizer by icebraining on Tuesday August 06, 2019 @07:45PM (#20630780)

There are multiple dedup utilities. I use fdupes, which can not only list but also remove or replace duplicates with hardlinks.

[ Parent | on HN ]

Re:Unison File Synchronizer by mindslight on Friday August 09, 2019 @11:58AM (#20655543)

Thanks! I guess formulating that question, I was halfway to apt-cache search. I had envisioned a tool that would find an entire equivalent dir tree (two unpacked source tarballs -> one match), and at first glance I don't know if fdupes will do that. But it might make sense to use an off the shelf tool with a little more manual labor.

[ Parent | on HN ]