News for hackers. Stuff that matters.
Re:Crawling the web at $2 per million pages by bliving on Wednesday September 23, 2009 @09:20PM (#840445)
That and...

"Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed"

... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.

There are no comments on this comment yet.