News for hackers. Stuff that matters.
Re:Crawling the web at $2 per million pages by mwexler on Wednesday September 23, 2009 @08:58PM (#840419)
One limitation is the throttling for each domain. If I had a smaller set of pages that I wanted to read frequently, this solution would not work. I understand the need to avoid DOS attacks, but in some cases, it would nice to be able to read a million pages from 3 large domains instead of 1 from a million pages.
Re:Crawling the web at $2 per million pages by bliving on Wednesday September 23, 2009 @09:20PM (#840445)
That and...

"Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed"

... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.