News for hackers. Stuff that matters.
Re:WARC'in the Crawler by electroly on Saturday December 23, 2023 @01:50AM (#38742219)
If you're creating the zips in the first place, you can just check and see how big the headers are when you create them. If you happen to get N wrong, you can request another chunk, but obviously it's nice to avoid multiple requests to get the header. For my use case, the number of files is small and relatively consistent between zips so a generous value of 64KB ended up working great.

There are no comments on this comment yet.