Content addressing is one of the big advantages of p2p applications, and IPFS has been pushing it for a long time.
Just imagine if this was done all the way down to the piece level. You have two different torrents (say, Linux ISOs) where 20% of the pieces overlap due to similarity (maybe a point upgrade or something and you want both versions). Rather than 100% of both, you only need to download the shared pieces once. Not only that, but say that the latest version has many more seeds/peers, you could download the pieces from that swarm instead, saving the bandwidth of the older torrent's swarm for the remaining 80% you need to download.
This could probably be done client side somehow, but it would be good to see actual protocol support for it so that bittorent can move in that direction.
The idea is to make blocks (slightly) variable in size. Block boundaries are determined based on a limited window of preceding bytes. This way a change in one location will only have a limited impact on the following blocks.
There isn't a way to advertise some "rolling hash value" in a way that allows other people with a differently-aligned copy to notice that you and them have some duplicated byte ranges.
Rolling hashes only work when one person (or two people engaged in a conversation, like rsync) already has both copies.
The rolling hash is used to find the chunk boundary: Hash a window before every byte (which is cheap with a rolling hash) and compare it against a defined bit mask. For example: Check if the first 20 bytes are zero. If so, you'd get chunks with about 2^20 bytes (1 MiB) average length.
As a good explanation, I'd encourage you to look at borgbackup's internals documentation: https://borgbackup.readthedocs.io/en/stable/internals.html
If I discover that the file I want to publish shares a range with an existing file, that does very little because the existing file has already chosen its chunk boundaries and I can’t influence those. That ship has sailed.
I can only benefit if the a priori chunks are small enough that some subset of the identified match is still addressable. And then I may only get half of a two thirds of the improvement I was after.
If they both used the same rolling hash function on the same or similar data, regardless of the initial and final boundary and regardless of when they chose the boundaries, they will share many chunks with high probability. That’s just how splitting with rolling hashes work. They produce variable-length chunks.
> Identical files will always have the same hash and can more easily be moved from one torrent to another (when creating torrents) without having to re-hash anything.
also need to watch out for compression
I don't know about that. With a reasonably large number of users, wouldn't you start seeing hash collisions via birthday paradox? Might be better off keeping it at the file level. (Though this does incentivize malicious users to find hash collisions of particular files they want to defend, and seeding the swarm with garbage files)
Does "get a collision" here mean finding a particular collision with a known hash, or finding any collision between any two of the generated hashes?
https://github.blog/2017-03-20-sha-1-collision-detection-on-...
sha-256 was chosen because it provides a sufficiently high number of different hashes that this won't be an issue for the forseeable future.
more details here: https://stackoverflow.com/questions/4014090/is-it-safe-to-ig...
Yes. It also makes your files more discoverable for copyright trolls. They will be able to fully automate the process of sending takedown notices for each copy of zlib's README.txt, that was "stolen" from their software by evil pirates.
Except for so many that are a single archive file. It seems logical that you'd want to compress what you're sending as much as possible before uploading... but in today's bandwidth environment, it actually makes sense to leave certain things uncompressed so you can benefit from swarm overlaps?!? Wild!
It's not uncommon to see multiple .torrent files with the same content FYI.
Yes, this won't be useful for speed.
But it will be useful for health and appearance of health.
So a torrent that has no seeds because it's missing "RARBG_DO_NOT_MIRROR.exe" can be filled in quickly and have proper seed info.
Also large seasons can be filled in. They get corrupted at later episodes because people will prioritise the first episodes then the seeds can jump.
So anyone filling in the later episodes through single episodes torrents might repair the season torrent.
It'll allow over lap between torrents only differentiated be a missing RARBG.txt if they are both fragmented.
There's always been talk of allowing a torrent to have files added to it. It's be interesting to see if this might be useful client side to do this in a Claytons way.
Personally I don't think it's useful for Linux isos. Not sure on this. Do Linux isos get unhealthy?
but it would take some handcrafting to make sure the new files are at the end of the ISO so everything else stays in place.
technically this should be possible since ISO should allow for adding files given that it was designed for write-once media.
> Files that are identical can also more easily be identified across different swarms, since their root hash only depends on the content of the file.
Wait, content addressed blobs across swarms... does that mean torrents made by completely different people at different times that happen to contain one or more identical files can benefit from each other's peers? If so this feels like a significant feature that would boost the long term health of a lot of torrents and connect more peers that could be helping each other.