Re:Common Crawl
by duskwuff on Friday March 26, 2021 @07:04PM
(#26598044)
As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months):
https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...