News for hackers. Stuff that matters.
Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @07:33PM (#39184811)
> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes.

I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.

Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @07:54PM (#39184960)
WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.

A CSV file is arguably harder to recover by comparison.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @08:16PM (#39185170)
> WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.

I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?

Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @08:30PM (#39185288)
It supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.