News for hackers. Stuff that matters.

A comment on How to Crawl the Web Politely with Scrapy. Parent comment

Re:How to Crawl the Web Politely with Scrapy by minimaxir on Thursday August 25, 2016 @11:23AM (#12359715)
In the context of this submission, it's definitely not polite.
[ Parent | on HN ]
Re:How to Crawl the Web Politely with Scrapy by emodendroket on Thursday August 25, 2016 @01:49PM (#12360986)
It's certainly more considerate if you limit your scraping to rates similar to a human user instead of just going as fast as possible.
[ Parent | on HN ]