News for hackers. Stuff that matters.
Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @06:50PM (#39184409)
> - Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly.

Filesystems don't generally deal well with having billions of files. Like you can work around some issues with deep directory structures, but even then it's not very effective and many cases you'll run out of inodes before you even reach a single billion. This is not a use-case filesystems in general tend to optimize for.

> - Why not use SQLite?

The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. There are much fewer things that can go wrong compared other formats. Records are only stored in exactly one location. Data corruption, write faults and other error states are all easy to detect and reason about. You have to work really hard to mess up a WARC file beyond being able to salvage most of it.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @07:33PM (#39184811)
> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes.

I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.

Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @07:54PM (#39184960)
WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.

A CSV file is arguably harder to recover by comparison.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @08:16PM (#39185170)
> WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.

I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?

Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @08:30PM (#39185288)
It supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.