News for hackers. Stuff that matters.
Re:WARC'in the Crawler by stavros on Friday December 22, 2023 @01:30PM (#38737000)
How can you guarantee that it's always within N kilobytes, when N depends on the number of files in the zip?
Re:WARC'in the Crawler by electroly on Saturday December 23, 2023 @01:50AM (#38742219)
If you're creating the zips in the first place, you can just check and see how big the headers are when you create them. If you happen to get N wrong, you can request another chunk, but obviously it's nice to avoid multiple requests to get the header. For my use case, the number of files is small and relatively consistent between zips so a generous value of 64KB ended up working great.