News for hackers. Stuff that matters.

A comment on How to Crawl the Web Politely with Scrapy. Parent comment

Re:How to Crawl the Web Politely with Scrapy by greglindahl on Thursday August 25, 2016 @12:15PM (#12360237)
Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.
[ Parent | on HN ]
Re:How to Crawl the Web Politely with Scrapy by stummjr on Thursday August 25, 2016 @12:30PM (#12360347)
That's kind of what Scrapy's AUTO_THROTTLE middleware does.
[ Parent | on HN ]