Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Web Scraping and Crawling Are Perfectly Legal, Right?. Parent comment
If you generate a load greater than what a human would do, this might become problematic. A human wouldn't poke a website forever every 10-15 seconds.
The 10-15 seconds is more for web crawlers. Eventually, a crawler will run out of pages to crawl on a website, so it will stop sending requests to it.
Personally, I would stick to the 3 minutes delay. But it depends on what type of website you're scraping :)
There are no comments on this comment yet.
If you generate a load greater than what a human would do, this might become problematic. A human wouldn't poke a website forever every 10-15 seconds.
The 10-15 seconds is more for web crawlers. Eventually, a crawler will run out of pages to crawl on a website, so it will stop sending requests to it.
Personally, I would stick to the 3 minutes delay. But it depends on what type of website you're scraping :)