News for hackers. Stuff that matters.
Re:Show HN: Semantics3 – API for Products and Prices by netvarun on Wednesday February 13, 2013 @11:17PM (#5217685)
Some great advice here on crawling at scale, which has inspired our crawlers a lot : http://news.ycombinator.com/item?id=4367933

Basically it boils down to three things: 1. If the site is slow,crawl slooowly. 2. If you see non-200 http error codes, stop! 3. Obey robots.txt and speed restrictions.

There are no comments on this comment yet.