100TB. Dang! I will need to expand my storage significantly to mirror it.

100TB so far ;)

And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet.

the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain.

it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.