News for hackers. Stuff that matters.

A comment on Common Crawl. Parent comment

Re:Common Crawl by new_realist on Friday March 26, 2021 @05:48PM (#26597369)
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior snapshots unless you're interested in historical content.

EDIT: the state of the crawls are summarized at https://commoncrawl.github.io/cc-crawl-statistics/.

[ Parent | on HN ]
Re:Common Crawl by duskwuff on Friday March 26, 2021 @07:04PM (#26598044)
As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months):

https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...

[ Parent | on HN ]