Re:How to Crawl the Web Politely with Scrapy
by stummjr on Thursday August 25, 2016 @09:24AM
(#12358778)
Scrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address.
And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.
And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.