Re:How to crawl a quarter billion webpages in 40 hours
by binarysolo on Saturday August 11, 2012 @07:58PM
(#4371355)
Gray hat question out of curiosity and possible experience: did you also use proxies or perhaps even Tor?
For me, it was more a proof of how efficient and fast a crawler can be. Also, a response from IMDB was very fast in less than 0.4 seconds, so not that much time was lost there.