Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Show HN: Semantics3 – API for Products and Prices. Parent comment
Basically it boils down to three things: 1. If the site is slow,crawl slooowly. 2. If you see non-200 http error codes, stop! 3. Obey robots.txt and speed restrictions.
There are no comments on this comment yet.
Basically it boils down to three things: 1. If the site is slow,crawl slooowly. 2. If you see non-200 http error codes, stop! 3. Obey robots.txt and speed restrictions.