Re:Duplicacy: Lock-free deduplication cloud backup tool, wit...
by acrosync on Wednesday June 07, 2017 @11:55AM
(#14507413)
Naming chunks by their hashes is not a new idea, but this technique along does not give you a practical backup tool. The deletion of unreferenced chunks becomes a hard problem, and the center piece of lock-free deduplication is the two-step fossil collection algorithm that solves this hard problem.
"It is the only cloud backup tool that allows multiple computers to back up to the same storage simultaneously without using any locks (thus readily amenable to various cloud storage services)"
"What is novel about lock-free deduplication is the absence of a centralized indexing database for tracking all existing chunks and for determining which chunks are not needed any more. Instead, to check if a chunk has already been uploaded before, one can just perform a file lookup via the file storage API using the file name derived from the hash of the chunk."
Tahoe-LAFS's immutable file model (based on convergent encryption) was capable of doing this same thing a decade ago, and also features a pretty nifty capability-based security model:
https://tahoe-lafs.org/trac/tahoe-lafs