How to crawl a quarter billion webpages in 40 hours

Re:How to crawl a quarter billion webpages in 40 hours by joshu on Saturday August 11, 2012 @01:40PM (#4370548)

so how polite does one need to be? One hit per x seconds?

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by yorhel on Sunday August 12, 2012 @03:51AM (#4372127)

If the /robots.txt does not mention a Crawl-delay, one page per 3 seconds is often a safe value. Of course this rather heavily depends on the site. In any case, if you have any specific need, always contact the people responsible for the site. I occasionaly run custom queries against the database on request, for example.

[ Parent | on HN ]