Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Let us serve you, but don't bring us down.
(1) Limit your load / parallelism: do not use more than 4 threads, and if the site takes more than a few seconds to respond, use fewer.
(2) Limit aggregate load: after each request, sleep/wait at least the amount that the previous request took to get served.
(3) If you need more than that, ask the site owners for direct channel.
This way if multiple crawler happen to crawl the same resource-limited site, the site may have a fighting chance.
There are no comments on this comment yet.
(1) Limit your load / parallelism: do not use more than 4 threads, and if the site takes more than a few seconds to respond, use fewer.
(2) Limit aggregate load: after each request, sleep/wait at least the amount that the previous request took to get served.
(3) If you need more than that, ask the site owners for direct channel.
This way if multiple crawler happen to crawl the same resource-limited site, the site may have a fighting chance.