I really like the IPFS solution to this. Basically a torrent where each chunk is addressable, so you get deduplication on each mirror even across unrelated file hierarchies.