Re:Crawling the web at $2 per million pages
by bliving on Wednesday September 23, 2009 @09:20PM
(#840445)
That and...
"Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed"
... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.
"Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed"
... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.