Re:BitTorrent v2 (2020)
by SethTro on Monday January 10, 2022 @01:42PM
(#29879457)
It is highly unlikely that the duplicate portions of the file will have an offset that's a multiple of 2^16 which would be required for chunks to have matching hashes. On the client side you could theoretically run lbfs over your files but on the swarm side this isn't going to happen
[ Parent
| on HN ]
Re:BitTorrent v2 (2020)
by Dylan16807 on Monday January 10, 2022 @01:58PM
(#29879779)
> It is highly unlikely that the duplicate portions of the file will have an offset that's a multiple of 2^16 which would be required for chunks to have matching hashes.
That's exactly what chunking based on a rolling hash solves. You set the average size of chunks and the content controls the exact boundaries.
Re:BitTorrent v2 (2020)
by gkfasdfasdf on Monday January 10, 2022 @06:53PM
(#29884909)
Right, exactly. Chunk boundaries are not determined by fixed size chunks, but rather when the rolling hash matches some prefix, which means chunk sizes will vary but by controlling the prefix can set the average size of the prefix. Besides the lbfs paper, another nice writeup here: https://moinakg.wordpress.com/2013/06/22/high-performance-co...
Re:BitTorrent v2 (2020)
by brokenmachine on Monday January 10, 2022 @07:29PM
(#29885309)
Rabin fingerprinting can do this I believe.
"the idea is to select blocks not based on a specific offset but rather by some property of the block contents"
https://en.wikipedia.org/wiki/Rabin_fingerprint#Applications