News for hackers. Stuff that matters.
Re:How to Crawl the Web Politely with Scrapy by stummjr on Thursday August 25, 2016 @09:24AM (#12358778)
Scrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address.

And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.

There are no comments on this comment yet.