News for hackers. Stuff that matters.
Re:Common Crawl by duskwuff on Friday March 26, 2021 @07:04PM (#26598044)
As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months):

https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...

There are no comments on this comment yet.