News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours (2012) by marginalia_nu on Thursday June 15, 2023 @11:15AM (#36341716)
I run a search engine crawler from my residential network. I get this too sometimes, but a lot of the time the IP shit-listing is temporary. It also seems to happen more often if you don't use a high enough crawl delay, ignore robots.txt, do deep crawls ignoring HTTP 429 errors and so on. You know, overall being a bad bot.

Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.

There are no comments on this comment yet.