News for hackers. Stuff that matters.
Re:Crawling the web at $2 per million pages by jdrock on Thursday September 24, 2009 @01:29AM (#840662)
While complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need several servers, IP addresses, and much more.

There are no comments on this comment yet.