News for hackers. Stuff that matters.
Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @06:02PM (#39183981)
Great article on the WARC file format. I really appreciate the efforts of the ArchiveIt team in developing and maintaining such a crucial tool for web archiving.

I'm trying to understand why the WARC format was made the way it is. Can someone explain the main reasons behind its design? Specifically, I'm curious about:

- Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly.

- Why not use SQLite?

Re:An Introduction to the WARC File by egh on Monday January 29, 2024 @06:38PM (#39184293)
It's a pretty simple design, and it's based on the ARC format (https://archive.org/web/researcher/ArcFileFormat.php) which is even simpler. In response to your questions, here's my take (as somebody who used to work on web archiving).

1. Two reasons: First, many files are harder to manage. WARC files might contain hundreds or thousands of files. It's easier to manage big groups of files that are roughly the same size. Both for humans, and, at least in the past, for the file systems themselves. Second, once you break them up into files, what do you name the files? If you give them a name unrelated to the URL that was fetched, what is the advantage? If you name them based on the URL, suddenly you have a problem of mapping a URL to a legal file name, which can vary based on the file system. This would be a huge headache.

2. Yes, it predates SQLite, but also, why would you use sqlite? That's adding a huge amount of complexity. Is SQLite even good at storing big binary blobs?

Additionally, because of the clever way that WARC files are gzipped, each piece of the WARC file is gzipped individually, which allows random access into the file for reading enclosed content in a compressed file without needing to read the entire WARC file.

Re:An Introduction to the WARC File by abracadaniel on Monday January 29, 2024 @06:51PM (#39184416)
To add to this, WARC.gz files are also concatenated gzip records, so you can read any record by starting a decompression at a known offset. This gives you the access time of a file with the efficiency of having many many records only taking up one file.
Re:An Introduction to the WARC File by nikisweeting on Tuesday January 30, 2024 @03:27AM (#39187656)
WACZ also extends this functionality to allow streaming archives off a server without having to request the whole file to get one page. https://replayweb.page/docs/wacz-format
Re:An Introduction to the WARC File by Retr0id on Monday January 29, 2024 @06:41PM (#39184325)
> Is SQLite even good at storing big binary blobs

Nope! SQLite is good for lots of small-ish blobs (kilobytes), but once you start getting into the megabyte range, less so. There's also currently a hard upper limit blob size of 2GiB.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @06:49PM (#39184403)
Thanks for the insights, egh! It's clear now why SQLite wouldn't be ideal for this purpose. Also, the point about URLs not always being valid filenames really makes sense.
Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @06:50PM (#39184409)
> - Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly.

Filesystems don't generally deal well with having billions of files. Like you can work around some issues with deep directory structures, but even then it's not very effective and many cases you'll run out of inodes before you even reach a single billion. This is not a use-case filesystems in general tend to optimize for.

> - Why not use SQLite?

The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. There are much fewer things that can go wrong compared other formats. Records are only stored in exactly one location. Data corruption, write faults and other error states are all easy to detect and reason about. You have to work really hard to mess up a WARC file beyond being able to salvage most of it.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @07:33PM (#39184811)
> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes.

I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.

Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @07:54PM (#39184960)
WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.

A CSV file is arguably harder to recover by comparison.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @08:16PM (#39185170)
> WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing.

I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?

Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @08:30PM (#39185288)
It supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.
Re:An Introduction to the WARC File by SnowflakeOnIce on Monday January 29, 2024 @06:09PM (#39184046)
As to the your second question: WARC is based on an older format that was started in the 1990s, before SQLite existed.
Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @06:26PM (#39184185)
Thanks for that info! Didn't realize WARC predated SQLite. Makes me wonder, are there any modern updates that could enhance WARC, or is it only format for archiving?
Re:An Introduction to the WARC File by makeworld on Monday January 29, 2024 @06:31PM (#39184242)
The main update to WARC now is WACZ.

https://specs.webrecorder.net/wacz/1.1.1/

Re:An Introduction to the WARC File by TheTechRobo on Wednesday January 31, 2024 @03:58PM (#39209319)
Yeah, the format really needs an update. For starters, WARC only officially supports HTTP/1.1. Webrecorder has started faking HTTP/1.1 data in WARC files in order to save other versions, but I don't think faking data is great for an archival format, especially if it isn't standardized.
Re:An Introduction to the WARC File by michaelt on Monday January 29, 2024 @07:12PM (#39184606)
> Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly.

First of all, no matter how easy it is to extract the files, the results will be kinda user unfriendly - unless someone goes through the archived html files updating all the URLs for images and javascript and so on, it'll probably end up looking pretty broken. For users who just want to view a few files, Wayback Machine is a better tool for the job.

With that said, some situations where warc is helpful include:

* If a website had a page at www.example.com/foo and a file at www.example.com/foo/bar.txt you can't express that in a regular filesystem, as you can't have a file and a directory with the same name.

* If a website uses some absurd 2000 character URL like https://s3.eu-west-2.amazonaws.com/document-api-images-live.... you don't end up with a filesystem-breaking filename.

* If www.example.com embeds an image from exampleusercontent.com you can capture the image in the same archive file.

* If for some reason you want to store daily copies of www.cnn.com in the same archive for comparison purposes - you can.

* It lets you store headers, 300 redirect messages, case-sensitive filenames, and all that sort of stuff.

* And it's an extremely simple format - basically human readable. So if you think you're archiving for the super-long-term and want to make really conservative choices, you can be pretty confident in plain text.

Re:An Introduction to the WARC File by kburman on Monday January 29, 2024 @08:12PM (#39185139)
Interesting points about the WARC format.I'm just wondering how it handles tricky stuff like those cache-busting query parameters and dynamic content generated by scripts. These seem like important details for preserving web pages accurately.
Re:An Introduction to the WARC File by marginalia_nu on Monday January 29, 2024 @10:01PM (#39185916)
It's basically like a dumb flight recorder. It saves the http request and the response, including all headers, cookies, and the full query string, with additional metadata on top.
Re:An Introduction to the WARC File by fforflo on Tuesday January 30, 2024 @03:55AM (#39187835)