(Update: the complete Common Crawl dataset is actually a little smaller than I thought, at 6.4 PB. That's still pretty big, though.)
Re:Common Crawl
by indolering on Friday March 26, 2021 @10:01PM
(#26599185)
> Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down.
Amazon makes plenty of money from people using AWS to processes the data.
That's a win-win.
Re:Common Crawl
by new_realist on Friday March 26, 2021 @05:48PM
(#26597369)
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior snapshots unless you're interested in historical content.
EDIT: the state of the crawls are summarized at https://commoncrawl.github.io/cc-crawl-statistics/.
As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months):
https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...
Storing a dataset of this size and making it available online is not inexpensive. Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down.