Re:Show HN: Semantics3 – API for Products and Prices
by blart on Wednesday February 13, 2013 @09:34PM
(#5217315)
I have done some scraping of the amazon.com previously, but they are pretty good at detecting bot's and shutting them down, how did you get around this problem when scraping millions of pages?
[ Parent
| on HN ]
Re:Show HN: Semantics3 – API for Products and Prices
by netvarun on Wednesday February 13, 2013 @11:17PM
(#5217685)
Some great advice here on crawling at scale, which has inspired our crawlers a lot : http://news.ycombinator.com/item?id=4367933
Basically it boils down to three things: 1. If the site is slow,crawl slooowly. 2. If you see non-200 http error codes, stop! 3. Obey robots.txt and speed restrictions.