Re:An Introduction to the WARC File
by ghukill on Monday January 29, 2024 @08:08PM
(#39185109)
If interested in WARC, recommend also checking out WACZ: https://specs.webrecorder.net/wacz/1.1.1/
[ Parent
| on HN ]
What's the point of WACZ? It appears to wrap a number of WARC files into a single zip, enabling Range requests to specific WARC files so it can be served by a passive file server. But why is that needed?
Re:An Introduction to the WARC File
by nikisweeting on Tuesday January 30, 2024 @03:23AM
(#39187624)
It's huge for being able to replay big WARC files in a browser without having to download the whole thing. (e.g. try loading a 700mb WARC from IPFS to visit one page within it, it's too slow to work as-is)
It's used extensively by the Browsertrix/Webrecorder.io projects (who's team pioneered the WACZ format) and a few other projects.
Oh I may have missed that part. So the WACZ (indexes?) can contains offsets into the WARC file itself to each individual page?
Re:An Introduction to the WARC File
by nikisweeting on Tuesday January 30, 2024 @08:58AM
(#39190150)
WACZ is a replacement for WARC that has the index with offsets built in.
But it uses warc files inside as the archive format. It seems weird to call it a replacement when the original is still present.
Re:An Introduction to the WARC File
by nikisweeting on Tuesday January 30, 2024 @06:23PM
(#39197410)
I just meant from a user's perspective it's a format that superseeds WARC. But internally, yes, one is an encapsulation format for the other.