News for hackers. Stuff that matters.
Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @12:17AM (#5930954)
> This whole NSA thing should be enough to push things like named data networking/content centric networking

You're damn right. It wasn't even a few days after this whole thing broke that I started working on a simple, distributed and automatic file synchronization program. No third parties and open source.

I was absolutely stunned to find out that nothing out there meets those criteria (despite my best efforts). The only thing that was close was git-annex assistant, but I don't want to use git for simple file synchronization (particularly for large files).

There's also Bittorrent Sync, but it's completely locked down. :-/

Re:Skype's Principal Architect explains why peer-to-peer was... by pthatcherg on Monday June 24, 2013 @01:45PM (#5934480)
FYI,

Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly) worked. We spent a lot of time trying to solve things that are not largely solved, so if I were to start over today, a lot of the hard work would be done.

To be more specific, if I were you, I would start by using the p2p connectivity and data channel code built into WebRTC (at webrtc.org) or libjingle (at https://code.google.com/p/libjingle/). This is what AeroFS (at https://aerofs.com/) is doing.

By the way, writing a distributed file sync system is not that simple, even though it might seem simple at first. Things get complicated really fast. I think that ones reason one of these hasn't taken off is because it's really hard to get right. But, I still hope it will happen one day.

Oh, and there is http://camlistore.org/, which is an open source project run by a really smart Googler. It's not specific to file syncing, but it has a powerful foundation. A combination of the p2p bits from WebRTC and the distributed data bits from camlistore would be pretty powerful. It makes me wish I could find some more 20% time :).

(Full disclosure: I work at Google, often on the WebRTC code base, and am the maintainer of libjingle)

Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @02:32PM (#5934767)
I can hardly consider myself an amateur with respect to distributed computing, so I am ignorant enough to believe that it is not that complex. Even worse, I won't believe it until I find out for myself. :-)

I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency.

Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail.

Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant.

[1] - http://camlistore.org/pkg/

Re:Skype's Principal Architect explains why peer-to-peer was... by pthatcherg on Monday June 24, 2013 @05:53PM (#5935967)
To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly. Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide.

By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.

Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @06:33PM (#5936185)
> To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly.

Those are exactly the things I intend to leave out in my initial work. If it happens to catch on, hopefully I'll get some help. If not, it will still be very useful to me without those features.

> Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide.

Yup. I think this is one of the things that Bittorrent Sync gets right: strong symmetric AES encrpytion.

> By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.

Not to write in it. In the Go community, there's a heavy bias to avoid linking to external libraries if it can be avoided. There is a certain amount of cachet that comes from being able to do `go get ...` and have it work without having to make sure another library is installed.

I'm at the infancy stages of my understanding, so it's certainly possible that I'll come to my senses and build on top of something else. But for now, I'm going to blaze my own trail. At least partially to learn. (Two of the most important pieces---encryption and p2p---are things I have very little experience with.)

Re:Skype's Principal Architect explains why peer-to-peer was... by fulafel on Tuesday June 25, 2013 @11:13AM (#5940000)
C++ free ICE implementations:

C: http://nice.freedesktop.org/wiki/ (glib-based so wrapping friendly to other gnome-supported languages)

Java: https://java.net/projects/siptools/pages/IcedJava

Re:Skype's Principal Architect explains why peer-to-peer was... by pwnna on Monday June 24, 2013 @12:29AM (#5930988)
This is all good but the biggest thing right now is that there is no coordinated effort to build a system that's easy to use, secure, and open source.

Everyone can spend their time working on their own solution but no one would use it as everyone's work will be incompatible with each other. This means no one (or rather, not enough people) will use any solutions that are developed and revert back to something like Skype, FB message, etc.

I think what we need is to establish a foundation that coordinates the development efforts for these secure and reliable software packages with a few centralized projects that become a standard (here's hope for WebRTC).

Re:Skype's Principal Architect explains why peer-to-peer was... by lukifer on Monday June 24, 2013 @02:55AM (#5931394)
> There's also Bittorrent Sync, but it's completely locked down.

This really surprised me; for some reason, I assumed it was open source. Since the storage is entirely peer-to-peer, I wonder what their business model is?

Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @02:59AM (#5931404)
Who knows. It's completely opaque to me. I just don't want to be left high-and-dry if the maintainers decide to abandon the project. Plus, I want to be able to tinker, of course. :-)

[EDIT] This was all I could find.[1] Which isn't much.

[1] - http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-...

Re:Skype's Principal Architect explains why peer-to-peer was... by mtgx on Monday June 24, 2013 @05:34AM (#5931903)
Not open source, probably because the bittorrent protocol isn't open source.
Re:Skype's Principal Architect explains why peer-to-peer was... by goblin89 on Monday June 24, 2013 @07:22AM (#5932195)
I believe that's not correct. First a nitpick, I think it's only software that can be open or closed source. Protocol specification might be kept secret, though.

Regarding specifically BitTorrent protocol specification[0], it's free to use. Anyone can build their open or closed source BitTorrent client without reverse engineering.

[0] http://www.bittorrent.org/beps/bep_0003.html

Re:Skype's Principal Architect explains why peer-to-peer was... by lukifer on Monday June 24, 2013 @05:56AM (#5931948)
How so? Several open-source clients exist. Were they reverse-engineered?
Re:Skype's Principal Architect explains why peer-to-peer was... by joshaidan on Monday June 24, 2013 @08:28AM (#5932429)
I believe the first bittorrent client was written in python. http://en.wikipedia.org/wiki/Bram_Cohen
Re:Skype's Principal Architect explains why peer-to-peer was... by icebraining on Monday June 24, 2013 @02:59AM (#5931403)
Just for checking: git-annex doesn't actually check the files into the git repository, so whether the file is small or large is irrelevant to it. Git is only used to keep track of the filenames and their locations.
Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @03:33AM (#5931502)
Interesting. The web site really makes it sound otherwise:[1]

    Then any changes you make to its folder will
    automatically be committed to git, and synced to
    repositories on other computers.
Nevertheless, I checked it out myself and you are indeed correct.

I guess that means I'm out of legitimate excuses? :P

[1] - http://git-annex.branchable.com/assistant/quickstart/

Re:Skype's Principal Architect explains why peer-to-peer was... by gioele on Monday June 24, 2013 @08:35AM (#5932458)
> Then any changes you make to its folder will automatically be committed to git

The _changes_ are tracked (addition, removals, etc.) not the content.

Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @10:56AM (#5933300)
Well, yes...... I know that now. But my initial reading did not catch that. Hence the confusion.
Re:Skype's Principal Architect explains why peer-to-peer was... by Fixnum on Monday June 24, 2013 @12:53AM (#5931063)
Out of curiosity, what's wrong with Unison?
Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @01:06AM (#5931118)
That's what I use now. But it's not automatic and is designed for use between two end points. I want it to be automatic across N end points. (And I intend to use a bittorrent-like protocol to achieve that, rather than an rsync-like protocol that unison uses.)

[EDIT] I forgot one of my most frustrating nits with unison: you need to have the same version on both ends. It's solvable by building the right version yourself, but it's damn annoying.

Re:Skype's Principal Architect explains why peer-to-peer was... by VLM on Monday June 24, 2013 @09:09AM (#5932617)
"I want it to be automatic across N end points."

I solved that problem, using unison, I guess near 15 years ago the unix way, not the windows way.

Its been awhile but with a little symlink fun you can make a little shell script that builds a partial mesh and syncs both the script itself, its list of possible mesh members (as opposed to the 3 or so I used) and all the other stuff, more or less silently run out of .bash_profile or .bashrc or whatever it was. There's already excellent tools in unix for scripting, automation, batch, etc, so no need to include them all into one tool.

The windows way, would be to put all those tools into one (buggy) executable with a gui and an installer that also installs browser toolbars or whatever.

I switched to git away from unison and never looked back. Eventually it got to be too big of a PITA to install 2 or 3 debian packages of different versions of unison all of which can only talk to certain other versions. So then my shell script had to detect which unison versions are installed on this node, and match up to the master list of other nodes and their versions, ... oh forget all this garbage just try this new subversion svn thing. And later, forget this whole script, just put everything important thats small in git and everything thats big in multiple AFS servers running on RAID.

Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @10:59AM (#5933318)
Yes, most of my stuff is kept in sync with git. But there are many things I want to keep in sync that I don't want to check into git. I'd rather they just sync automatically without intervention from me.
Re:Skype's Principal Architect explains why peer-to-peer was... by erikpukinskis on Monday June 24, 2013 @03:38AM (#5931520)
Wouldn't it be easier to contribute those changes to Unison instead of starting over?
Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @03:41AM (#5931530)
Dunno. They feel like pretty big changes to me. I imagine it'd have to be completely gutted since the entire protocol would have to change.

Plus, I am not fond at all of its interface/configuration. Very complex.

Re:Skype's Principal Architect explains why peer-to-peer was... by whadar on Monday June 24, 2013 @05:16AM (#5931851)
You can try the open sourced project, webrtc based, Sharefest.me for filesharing p2p with no cloud storage...
Re:Skype's Principal Architect explains why peer-to-peer was... by davedx on Monday June 24, 2013 @09:08AM (#5932609)
There's a WebRTC demo that does something like this. I can't seem to find the link, but it was linked from one of the official/reference implementation sites.
Re:Skype's Principal Architect explains why peer-to-peer was... by Scaevolus on Monday June 24, 2013 @04:10PM (#5935343)
Have you looked into CurveCP? It would provide security and automatic bandwidth throttling with minimal effort.
Re:Skype's Principal Architect explains why peer-to-peer was... by endgame on Monday June 24, 2013 @02:10AM (#5931299)
Your synchronisation project sounds intersting, do you have source code online yet?
Re:Skype's Principal Architect explains why peer-to-peer was... by burntsushi on Monday June 24, 2013 @02:34AM (#5931351)
Not yet. I have a piece or two on my github, but otherwise it hasn't been written yet. I'll be sure to post here when I'm done, but I've never had much luck with that in the past.