Re:Crawling the web at $2 per million pages
by jdrock on Thursday September 24, 2009 @01:29AM
(#840662)
While complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need several servers, IP addresses, and much more.
[ Parent
| on HN ]
There are no comments on this comment yet.