News for hackers. Stuff that matters.

A comment on 90% of Claude-linked output going to GitHub repos w <2 stars. Parent comment

Re:90% of Claude-linked output going to GitHub repos w <2 stars by ccgreg on Thursday March 26, 2026 @10:00PM (#47538289)
> Common Crawl, with over one billion, nine hundred and seventy thousand web pages in their archive: 345TB.

Common Crawl is 300 billion webpages and 10 petabytes. I suppose your number is 1 of our 122 crawls.

[ Parent | on HN ]
Re:90% of Claude-linked output going to GitHub repos w <2 stars by genewitch on Thursday March 26, 2026 @11:16PM (#47538762)
oh, i didn't see that the 1.97 billion pages were crawled in a 11 day period earlier this month. either way, nearly 2,000,000,000 pages fit in ~third of a petabyte...

p.s. thanks for correcting me, i was using this information for something else, and now it's correct!

[ Parent | on HN ]