News for hackers. Stuff that matters.
Re:ClearSkies – open-source file syncing without cloud by nl on Monday February 24, 2014 @04:57AM (#7289949)
Works well for me. For those who want to try it, this is what I did.

Create a Dockerfile:

  # Ubuntu 12.04 + Python + Git
  FROM nlothian/python-git

  # Ruby
  RUN apt-get -y install libgnutls26 ruby1.9.1
  RUN apt-get -y install ruby1.9.1-dev

  RUN gem install rb-inotify ffi

  RUN git clone https://github.com/jewel/clearskies
Build:

  sudo docker build -t clearskies .
Run:

  sudo docker run -i -t clearskies /bin/bash
Then:

  mkdir /testdir
  echo 'testing' > afile
  cd /clearskies
  ./clearskies start
  ./clearskies share /testdir
That will print something out in the form:

  clearskies:SYNCXXXXXXXXXXXXXXXXXXXXXXXX
Note this, then start another clearskies docker container.

In that one:

  mkdir /testdir
  cd /clearskies
  ./clearskies start
  ./clearskies attach clearskies:SYNCXXXXXXXXXXXXXXXXX /sharedir
Wait a few seconds, and your file should appear.
Re:ClearSkies – open-source file syncing without cloud by nl on Monday February 24, 2014 @07:12PM (#7294287)
Hmm.

There do seem to be problems with some firewall scenarios (or at least I presume that is what it is..)

When I setup a node at home and one on an Azure box I see them discover each other, but no files seem to get copied.

Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @11:41PM (#7295624)
If you'll open an issue on github and attach the logs from both peers, I can look at it.
Re:ClearSkies – open-source file syncing without cloud by maho on Monday February 24, 2014 @03:46AM (#7289741)
I like that "untrusted peers" are defined in the protocol, but unfortunately it's only an optional addendum [1]. To me, untrusted peering is the most important feature and I hope that not being defined in the main protocol does not mean it will be step-mothered in the implementation.

BitTorrent Sync only supports untrusted peers via API [2], and the only other open-source BitTorrent Sync alternative that I am aware of [3] left it out completely.

[1] https://github.com/jewel/clearskies/blob/master/protocol/unt...

[2] http://www.bittorrent.com/intl/de/sync/developers/api

[3] https://github.com/calmh/syncthing/wiki

Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:07PM (#7291873)
At one point I had it as a required part of the protocol, but later decided that it added a lot of complexity so moved it to an extension.

I'm currently working on a minor reorganization of the protocol in the protocol_cleanup branch. I'll see if I can fold untrusted mode back into the core protocol.

Re:ClearSkies – open-source file syncing without cloud by atmosx on Monday February 24, 2014 @05:27AM (#7290048)
What exactly is the function of untrusted peers? Speed up the sync?
Re:ClearSkies – open-source file syncing without cloud by jffry on Monday February 24, 2014 @07:29AM (#7290407)
To use a friend's machine as an off-site secondary backup. They can store encrypted data, but cannot view or push changes to the data. I'm guessing.
Re:ClearSkies – open-source file syncing without cloud by toomuchtodo on Monday February 24, 2014 @12:26PM (#7292014)
If its untrusted, how can it be a canonical reference?
Re:ClearSkies – open-source file syncing without cloud by scott_karana on Monday February 24, 2014 @03:11PM (#7293123)
Cryptographic hashes and signatures, I presume?
Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:10PM (#7291888)
My original intent was that you could have it running on a linode or other virtual machine where you don't trust the hardware.
Re:ClearSkies – open-source file syncing without cloud by maho on Monday February 24, 2014 @07:37AM (#7290428)
Internet connections can be quite asymmetrical. I have a 32 MBit/s downlink, but only 1 MBit/s up. That makes updating data on my tablet with data from home painfully slow unless I am at home. For me, a fast peer that I don't need to trust is pretty helpful.

But apart from speed and redundancy, I also hope for economy of scale. If there were a small market where several hosters offer peered hosts with X GB for $Y/month, it could drive costs down for everyone. Dropbox is asking for $0.10/GB/mon, which is about twice as high as it could be if the market were efficient.

Re:ClearSkies – open-source file syncing without cloud by nacs on Monday February 24, 2014 @06:13AM (#7290174)
Likely also for data redundancy if you only have 1 computer (or if all your computers/peers are in one building the untrusted peers could be your 'offsite backup' of sorts).
Re:ClearSkies – open-source file syncing without cloud by rahimnathwani on Monday February 24, 2014 @03:42AM (#7289722)
How do the peers find each other? The installation instructions suggest that I just need to run 'clearskies share ...' on one computer, and 'clearskies attach ...' on the second computer. How will they find each other (on a LAN or, especially, if they're each behind different NAT routers)?

I see that there is some 'tracker' code in the repo:https://github.com/jewel/clearskies/tree/master/tracker

Must I run that somewhere that both computers can access, and tell them its address?

Re:ClearSkies – open-source file syncing without cloud by e12e on Monday February 24, 2014 @04:12AM (#7289823)
Looks like there are various modes of discovery, see under "Peer discovery":

https://github.com/jewel/clearskies/blob/master/protocol/cor...

I can't find any references to DHT in the code (but the protocol lists that as an extension).

For lan udp broadcast:

https://github.com/jewel/clearskies/blob/master/lib/broadcas...

For tracker client (apparently gets a list of tracker URIs from the config):

https://github.com/jewel/clearskies/blob/master/lib/tracker_...

https://github.com/jewel/clearskies/blob/master/lib/conf.rb

Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:02PM (#7291831)
There's a common tracker, currently running at clearskies.tuxng.com.

The plan is to add DHT support similar to the DHT used by BitTorrent. We'll seed the DHT using the tracker.

If you want you can also run your own tracker. You should also be able to add peers manually by IP address and port, but that ability is missing from the ruby client.

Re:ClearSkies – open-source file syncing without cloud by jedc on Monday February 24, 2014 @08:51AM (#7290690)
Another way to do file syncing without a particular cloud provider is Camlistore: http://camlistore.org/

Brad Fitzpatrick is one of the creators (LiveJournal, memcache, etc.) and it's rapidly getting better and better. That said, it can still be a bit tricky to get everything set up.

Re:ClearSkies – open-source file syncing without cloud by MoosePlissken on Monday February 24, 2014 @04:32PM (#7293529)
Camlistore looks really cool, thanks for posting it. I've been thinking something like this should exist for some time now.
Re:ClearSkies – open-source file syncing without cloud by skimmas on Monday February 24, 2014 @06:02AM (#7290146)
Finally... :) I've been waiting for a project like this to appear for ages... thankyou OSS developer angels.
Re:ClearSkies – open-source file syncing without cloud by PeterSmit on Monday February 24, 2014 @03:29AM (#7289684)
This is excellent. I have been waiting for an open source btsync clone.
Re:ClearSkies – open-source file syncing without cloud by tiedemann on Monday February 24, 2014 @07:29AM (#7290408)
Re:ClearSkies – open-source file syncing without cloud by thejosh on Monday February 24, 2014 @03:35AM (#7289701)
gitannex? :)
Re:ClearSkies – open-source file syncing without cloud by rquirk on Monday February 24, 2014 @03:55AM (#7289770)
Have you been able to use it? It is less user friendly than the proprietary solutions. There's tons of documentation, but nothing that says "here's how to use it like bittorrent sync". In fact IIRC the docs specifically say somewhere "you cannot use this like drop box".

Just sharing stuff between 2 PCs was very difficult (or I couldn't figure it out) and the annex program sat at 100% CPU most of the time doing nothing. Being written in Haskell is a turn off too. If I have to fix something, I want C, python, etc, not this crazy write-only language :-)

Re:ClearSkies – open-source file syncing without cloud by icebraining on Monday February 24, 2014 @05:17AM (#7290009)
Have you tried the Assistant and its web UI? It comes bundled with git-annex: http://git-annex.branchable.com/assistant/
Re:ClearSkies – open-source file syncing without cloud by rquirk on Monday February 24, 2014 @08:15AM (#7290550)
I did. Maybe I should give it another shot.

I recall my problem was trying to understand how to sync files I already had in other directories. Things certainly were not sync'd automatically. In the walkthrough it says you need to git-add files and then git-commit them http://git-annex.branchable.com/walkthrough/#index3h2

The ~/annex/ directory ended up with symlinks to git objects and the files themselves are nowhere to be found. I didn't know where things were/weren't sync'd already. Nothing sync'd across and the assistant just said "all done" or something similar. At one point I remeber it just containing a bunch of broken symlinks. Good job I was just testing it out, imagine if it replaced my actual files with broken symlinks.

What all this boils down to is that git-annex is not as fool-proof as the proprietary solutions claim to be and something equally Free as g-a, but less complicated, would be great.

Re:ClearSkies – open-source file syncing without cloud by Touche on Monday February 24, 2014 @08:27AM (#7290593)
I have. It kind-of worked but eventually corrupted my files. They all became test files with a long .git/ path in them.

The Jabber plugin doesn't work so it wasn't distributed like btsync. I had a central "server". What this mean is if computer A kicked off a sync, a node wouldn't get updated right away. The "server" doesn't automatically push to all of the clients.

It worked ok, I've tried just about ever sync solution out there. btsync is the simplest "just works" that I've found. The only problem I've found with btsync is that sometimes the mobile apps appear to be offline. But once they are woke up they'll start syncing.

btsync isn't a backup solution though, so I have bakthat backing up to Amazon Glacier.

Re:ClearSkies – open-source file syncing without cloud by aroch on Monday February 24, 2014 @12:45PM (#7292151)
It doesn't do homedir syncing under OSX, making it all but useless for me.
Re:ClearSkies – open-source file syncing without cloud by rahimnathwani on Monday February 24, 2014 @03:43AM (#7289725)
Re:ClearSkies – open-source file syncing without cloud by beagle3 on Monday February 24, 2014 @08:18AM (#7290560)
The JSON packets are limited to 16MB - if this is supposed to contain a manifest of a deep-directory-with-lots-of-files, that might not be enough. I regularly rsync (on a lan) trees of a million files and 8 levels deep. The manifest for such a configuration will not fit within 16MB.

I see there's an "rdiff manifest" extension, which is cool for syncing later changes - but the initial manifest will have to be transferred some other way.

Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:04PM (#7291851)
This is handled by the protocol; see https://github.com/jewel/clearskies/blob/master/protocol/cor....

As an aside, I am currently adding a more sophisticated manifest exchange in the "protocol_cleanup" branch that will remove the need to keep sending the entire manifest (other than on the first connection).

Re:ClearSkies – open-source file syncing without cloud by polskibus on Monday February 24, 2014 @04:43AM (#7289911)
I use nas4free + samba/CIFS daily, works like a charm. On each client (mobile, PC) I have an application that performs regular backups. In terms of alternatives, there's also OwnCloud that can be deployed in-house.

How does ClearSkies compare to existing private cloud solutions?

Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:44PM (#7292147)
ClearSkies doesn't require a central server like OwnCloud, it's peer-to-peer.
Re:ClearSkies – open-source file syncing without cloud by slavko on Monday February 24, 2014 @12:28PM (#7292033)
Incredible! The OP is using SQLite, particularly, Fossil, http://fossil-scm.org/. It upsets me that this this open source venture does not give credit where it is due. If I a wrong, please correct me.
Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:53PM (#7292206)
I'm confused. The protocol spec doesn't mention SQLite. The ruby proof-of-concept doesn't use SQLite. I've not seen fossil before that I can recollect (I am the author of the clearskies protocol spec).
Re:ClearSkies – open-source file syncing without cloud by jmspring on Monday February 24, 2014 @05:04AM (#7289976)
Granted, I'm on mobile, but how does this differ from BTSync for two computers? There needs to be some agent proxying discovery if machines aren't on the same local network.
Re:ClearSkies – open-source file syncing without cloud by p4bl0 on Monday February 24, 2014 @06:50AM (#7290277)
I think the goal is to have a free (as in freedom) BTSync.
Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:15PM (#7291932)
Peer discovery is the same as with btsync. It uses a central tracker, but you can also add peers manually. We're planning on adding DHT support.
Re:ClearSkies – open-source file syncing without cloud by tjaerv on Monday February 24, 2014 @07:14AM (#7290364)
Potentially interesting, but my interest waned considerably when I saw it's GPLv3.
Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:12PM (#7291909)
We're changing to LGPLv3. See https://github.com/jewel/clearskies/blob/master/license-chan....

The C++ implementation is also LGPL.

Re:ClearSkies – open-source file syncing without cloud by alexchamberlain on Monday February 24, 2014 @01:27PM (#7292457)
How did you reach this decision?

IMHO such licences actually hold free software back.

Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @02:10PM (#7292750)
You can read the entire thread here:

https://groups.google.com/forum/#!msg/clearskies-dev/sTlXzBO...

For the sync app itself GPL would have worked great, but we want to have an easy-to-integrate sync library. Hopefully this will reduce the number of apps that require a cloud service to be able to synchronize the user data between devices.

Re:ClearSkies – open-source file syncing without cloud by welly on Monday February 24, 2014 @07:49AM (#7290469)
Really? Because of the license, you're not interested in this project?

I find that very, very bizarre. Can you explain your thinking?

Re:ClearSkies – open-source file syncing without cloud by levosmetalo on Monday February 24, 2014 @08:08AM (#7290525)
Can _you_ explain why do _you_ think someone loosing interest in the project because of its license choice is bizarre?

What makes _your_ "I don't care about licensing, and neither should you" opinion not bizarre, and his "I don't care about projects with wrong license" bizarre?

I agree that parent comment didn't add much to the discussion, but that's not the reason to imply bizarreness of it.

Re:ClearSkies – open-source file syncing without cloud by welly on Monday February 24, 2014 @09:35AM (#7290956)
> Can _you_ explain why do _you_ think someone loosing interest in the project because of its license choice is bizarre

If it's a project that interests or is useful to me, then personally that trumps the license. If I discover a project useful to my workflow then I'll find a way to work with it, despite the license.

> What makes _your_ "I don't care about licensing, and neither should you" opinion not bizarre

Because it's my opinion and I generally don't find my opinions bizarre, otherwise I wouldn't hold them.

And I didn't say "I don't care about licensing, and neither should you" or even insinuate that.

He's more than welcome to care more about a license over a usefulness of piece of software and I'll still find it odd.

Re:ClearSkies – open-source file syncing without cloud by euank on Monday February 24, 2014 @08:15AM (#7290551)
Not him, but I can see a few good reasons.

GPL is rather limiting in how you can use the code. A large company (e.g. Apple) won't let something gpl be used heavily internally if they can help it since they won't be able to apply patches or modify it without releasing these changes ... this obligation adds significant legal burden and, furthermore, releasing the changes could reveal private details about the companies internals.

I don't think that the commentor's reason is a good one, but I can understand the viewpoint; GPLv3 is quite limiting for some uses.

Re:ClearSkies – open-source file syncing without cloud by chrj on Monday February 24, 2014 @11:27AM (#7291614)
As long as it's only used and distributed internally, I don't believe the GPL has a problem with them modifying it without disclosing their changes. That's my understanding.
Re:ClearSkies – open-source file syncing without cloud by indeyets on Monday February 24, 2014 @11:36AM (#7291671)
not really. they won't be able to publish product which incorporates GPL library without releasing this product under compatible license.

they can use and patch GPL product internally as long as they want.

Re:ClearSkies – open-source file syncing without cloud by wazoox on Monday February 24, 2014 @12:11PM (#7291901)
Indeed, AGPL v3 would have been way better.
Re:ClearSkies – open-source file syncing without cloud by atmosx on Monday February 24, 2014 @03:41AM (#7289716)
Why does the author wants to port the daemon from ruby to C++? Is it speed?
Re:ClearSkies – open-source file syncing without cloud by jewel on Monday February 24, 2014 @12:20PM (#7291975)
Speed isn't actually too much of an issue with the ruby client, since all the CPU time is spent in the GnuTLS and Digest::SHA256 code, which is already written in C.

The problem is portability to android and iOS. We additionally want to make the core easy to embed in other applications.

Re:ClearSkies – open-source file syncing without cloud by beagle3 on Monday February 24, 2014 @07:50AM (#7290473)
He mentions an android client. If you want it to be fast, or not drain your battery, you need Java or C++. If you want it to run on iOS or Windows Phone, C++ it is.
Re:ClearSkies – open-source file syncing without cloud by silasb on Monday February 24, 2014 @07:51AM (#7290478)
To make it easier to port to Android and IOS is my guess.
Re:ClearSkies – open-source file syncing without cloud by sarreph on Monday February 24, 2014 @10:51AM (#7291412)
I love the name for this. :)