News for hackers. Stuff that matters.
Re:Common Crawl by duskwuff on Friday March 26, 2021 @04:05PM (#26596310)
You may not grasp just how large the Common Crawl dataset is. It's been growing steadily at 200-300 TB per month for the last few years. I'm not certain how large the entire corpus is at this point, but it's almost certainly in the tens to low hundreds of petabytes. (This is significantly larger than the capacity of the entire Sia network, for example.)

Storing a dataset of this size and making it available online is not inexpensive. Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down.

Re:Common Crawl by duskwuff on Friday March 26, 2021 @06:56PM (#26597987)
(Update: the complete Common Crawl dataset is actually a little smaller than I thought, at 6.4 PB. That's still pretty big, though.)
Re:Common Crawl by indolering on Friday March 26, 2021 @10:01PM (#26599185)
> Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down.

Amazon makes plenty of money from people using AWS to processes the data.

Re:Common Crawl by fjeifisjf on Saturday March 27, 2021 @08:47AM (#26601925)
That's a win-win.
Re:Common Crawl by new_realist on Friday March 26, 2021 @05:48PM (#26597369)
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior snapshots unless you're interested in historical content.

EDIT: the state of the crawls are summarized at https://commoncrawl.github.io/cc-crawl-statistics/.

Re:Common Crawl by duskwuff on Friday March 26, 2021 @07:04PM (#26598044)
As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months):

https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...