News for hackers. Stuff that matters.
Re:Syncthing: Open Source Dropbox and BitTorrent Sync Replac... by lugg on Monday May 12, 2014 @06:39PM (#7735809)
I dont get it, why is deduping at a chunk level important, can't it just take it as a tradeoff and dedupe at the file level, or does that have the same problems? And why exactly is deduping important for a personal use case, bandwidth is relatively cheap.

In any case, while I want security in these systems to avoid mass survey style attacks, I assume anything I put over the wire to be compromised at least potentially. I keep the kind of files that need that level of encryption on encrypted flash drives.

Re:Syncthing: Open Source Dropbox and BitTorrent Sync Replac... by enscr on Tuesday May 13, 2014 @02:07AM (#7736731)
If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. They benefit from block level deduplication. They can be large in size.
Re:Syncthing: Open Source Dropbox and BitTorrent Sync Replac... by icebraining on Tuesday May 13, 2014 @02:12AM (#7736740)
If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers.

That doesn't follow - you can solve the problem without containers, e.g. by encrypting or hashing the filenames.

Containers protect the file sizes, but they lose the ability to selectively sync certain files to certain nodes.

Re:Syncthing: Open Source Dropbox and BitTorrent Sync Replac... by enscr on Tuesday May 13, 2014 @09:03AM (#7737897)
Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy.

Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders.

I'm not saying that one is better than other, just giving an example of where de-duping at block level is needed as @lugg asked.

Re:Syncthing: Open Source Dropbox and BitTorrent Sync Replac... by icebraining on Tuesday May 13, 2014 @10:02AM (#7738170)
Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy.

Sure; I was just talking about the file names. You can avoid the hierarchy part by transmitting it separately from the files in a secure way, though.

Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders.

git-annex does. You can implement rules to decide what files should be synced, which should be dropped locally, etc.

For example, an MP3 player running it might have a rule such as

  filename(*.mp3) or filename(*.ogg)
and it'll only sync those files, regardless of their directories.