News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by mey on Friday August 10, 2012 @04:52PM (#4368192)
Is there any reason you crawled for 40 hours? Was this the optimal cost from Amazon? Why not do this over more time using bid for spot instances?
Re:How to crawl a quarter billion webpages in 40 hours by waterlesscloud on Friday August 10, 2012 @05:51PM (#4368449)
He mentions spot instances near the end.

tl;dr - He didn't think about it until just before launching the experiment and worried that it would take too much time to understand the implications of changing his approach. Though he estimates it may represent a factor of five decrease in price.