Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly) worked. We spent a lot of time trying to solve things that are not largely solved, so if I were to start over today, a lot of the hard work would be done.
To be more specific, if I were you, I would start by using the p2p connectivity and data channel code built into WebRTC (at webrtc.org) or libjingle (at https://code.google.com/p/libjingle/). This is what AeroFS (at https://aerofs.com/) is doing.
By the way, writing a distributed file sync system is not that simple, even though it might seem simple at first. Things get complicated really fast. I think that ones reason one of these hasn't taken off is because it's really hard to get right. But, I still hope it will happen one day.
Oh, and there is http://camlistore.org/, which is an open source project run by a really smart Googler. It's not specific to file syncing, but it has a powerful foundation. A combination of the p2p bits from WebRTC and the distributed data bits from camlistore would be pretty powerful. It makes me wish I could find some more 20% time :).
(Full disclosure: I work at Google, often on the WebRTC code base, and am the maintainer of libjingle)
I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency.
Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail.
Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant.
By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.
Those are exactly the things I intend to leave out in my initial work. If it happens to catch on, hopefully I'll get some help. If not, it will still be very useful to me without those features.
> Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide.
Yup. I think this is one of the things that Bittorrent Sync gets right: strong symmetric AES encrpytion.
> By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.
Not to write in it. In the Go community, there's a heavy bias to avoid linking to external libraries if it can be avoided. There is a certain amount of cachet that comes from being able to do `go get ...` and have it work without having to make sure another library is installed.
I'm at the infancy stages of my understanding, so it's certainly possible that I'll come to my senses and build on top of something else. But for now, I'm going to blaze my own trail. At least partially to learn. (Two of the most important pieces---encryption and p2p---are things I have very little experience with.)
C: http://nice.freedesktop.org/wiki/ (glib-based so wrapping friendly to other gnome-supported languages)
Everyone can spend their time working on their own solution but no one would use it as everyone's work will be incompatible with each other. This means no one (or rather, not enough people) will use any solutions that are developed and revert back to something like Skype, FB message, etc.
I think what we need is to establish a foundation that coordinates the development efforts for these secure and reliable software packages with a few centralized projects that become a standard (here's hope for WebRTC).
This really surprised me; for some reason, I assumed it was open source. Since the storage is entirely peer-to-peer, I wonder what their business model is?
[EDIT] This was all I could find.[1] Which isn't much.
[1] - http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-...
Regarding specifically BitTorrent protocol specification[0], it's free to use. Anyone can build their open or closed source BitTorrent client without reverse engineering.
Then any changes you make to its folder will
automatically be committed to git, and synced to
repositories on other computers.
Nevertheless, I checked it out myself and you are indeed correct.I guess that means I'm out of legitimate excuses? :P
The _changes_ are tracked (addition, removals, etc.) not the content.
[EDIT] I forgot one of my most frustrating nits with unison: you need to have the same version on both ends. It's solvable by building the right version yourself, but it's damn annoying.
I solved that problem, using unison, I guess near 15 years ago the unix way, not the windows way.
Its been awhile but with a little symlink fun you can make a little shell script that builds a partial mesh and syncs both the script itself, its list of possible mesh members (as opposed to the 3 or so I used) and all the other stuff, more or less silently run out of .bash_profile or .bashrc or whatever it was. There's already excellent tools in unix for scripting, automation, batch, etc, so no need to include them all into one tool.
The windows way, would be to put all those tools into one (buggy) executable with a gui and an installer that also installs browser toolbars or whatever.
I switched to git away from unison and never looked back. Eventually it got to be too big of a PITA to install 2 or 3 debian packages of different versions of unison all of which can only talk to certain other versions. So then my shell script had to detect which unison versions are installed on this node, and match up to the master list of other nodes and their versions, ... oh forget all this garbage just try this new subversion svn thing. And later, forget this whole script, just put everything important thats small in git and everything thats big in multiple AFS servers running on RAID.
Plus, I am not fond at all of its interface/configuration. Very complex.
You're damn right. It wasn't even a few days after this whole thing broke that I started working on a simple, distributed and automatic file synchronization program. No third parties and open source.
I was absolutely stunned to find out that nothing out there meets those criteria (despite my best efforts). The only thing that was close was git-annex assistant, but I don't want to use git for simple file synchronization (particularly for large files).
There's also Bittorrent Sync, but it's completely locked down. :-/