News for hackers. Stuff that matters.

A comment on Hyperdrive – a novel file sharing network. Parent comment

Re:Hyperdrive – a novel file sharing network by mindslight on Wednesday December 09, 2015 @05:28PM (#10707234)
rsync and Rabin chunking are both based on rolling hashes determining where exactly to cryptohash. rsync operates "in place" and benefits from chunks being in consistent order, whereas Rabin chunking uses precomputation to draw from unordered basis chunks (eg cross-file chunking).
[ Parent | on HN ]
Re:Hyperdrive – a novel file sharing network by zbjornson on Wednesday December 09, 2015 @10:39PM (#10708720)
Pardon my ignorance, but I'm still not quite understanding the applications and benefits. Distributed file hosting? High-concurrency reads? Torrent-like? "file sharing network" is a bit ambiguous to me.

(I seem to have broke the demo with a 100 MB file, btw.)

[ Parent | on HN ]
Re:Hyperdrive – a novel file sharing network by mindslight on Thursday December 10, 2015 @12:15PM (#10711792)
(Not my project. I'm just commenting on the general techniques)

Rabin chunking should get you quick cross-file partial-file deduplication. So if you're already storing a file and then download a modified version (from someone else) with changed metadata in the header, only the differing data should be transferred/stored.

[ Parent | on HN ]