Also see the discussion about Bup 3 days ago: https://news.ycombinator.com/item?id=9074114
I really like git annex since the repository is still completely git compatible.
I really like git annex since the repository is still completely git compatible.
https://github.com/bup/bup strategically breaks files to chunks of ~8k length, in a way that makes small changes to a big file become a small change to a small number of chunks (ideally, changing one byte in a 80GB file would change exactly one such 8KB chunk -- and that is often the case, though it's common than 3 or 4 would change).
bup then puts[0] the chunks into git, together with a "reconstruction map". As a result, you can efficiently put huge files in git and only pay for the real deltas in storage.
[0] It doesn't actually use git or libgit - it writes git packs directly.