Re:How to crawl a quarter billion webpages in 40 hours (2012)
by benfrederickson on Thursday July 05, 2018 @01:45PM
(#17465226)
I analyzed the top 1 million robots.txt files looking for sites that allow google and block everyone else here: https://www.benfrederickson.com/robots-txt-analysis/ - it's a relatively common pattern for major websites
[ Parent
| on HN ]
There are no comments on this comment yet.