Re:An Introduction to the WARC File
by marginalia_nu on Monday January 29, 2024 @07:54PM
(#39184960)
WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.
A CSV file is arguably harder to recover by comparison.
> WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.
I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?
Re:An Introduction to the WARC File
by marginalia_nu on Monday January 29, 2024 @08:30PM
(#39185288)
It supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.
I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.