News for hackers. Stuff that matters.

A comment on BitTorrent v2 (2020). Parent comment

Re:BitTorrent v2 (2020) by Dylan16807 on Monday January 10, 2022 @01:58PM (#29879779)
> It is highly unlikely that the duplicate portions of the file will have an offset that's a multiple of 2^16 which would be required for chunks to have matching hashes.

That's exactly what chunking based on a rolling hash solves. You set the average size of chunks and the content controls the exact boundaries.

[ Parent | on HN ]
Re:BitTorrent v2 (2020) by gkfasdfasdf on Monday January 10, 2022 @06:53PM (#29884909)
Right, exactly. Chunk boundaries are not determined by fixed size chunks, but rather when the rolling hash matches some prefix, which means chunk sizes will vary but by controlling the prefix can set the average size of the prefix. Besides the lbfs paper, another nice writeup here: https://moinakg.wordpress.com/2013/06/22/high-performance-co...
[ Parent | on HN ]