Re:Crawling the web at $2 per million pages
by mwexler on Wednesday September 23, 2009 @08:58PM
(#840419)
One limitation is the throttling for each domain. If I had a smaller set of pages that I wanted to read frequently, this solution would not work. I understand the need to avoid DOS attacks, but in some cases, it would nice to be able to read a million pages from 3 large domains instead of 1 from a million pages.
[ Parent
| on HN ]
Re:Crawling the web at $2 per million pages
by bliving on Wednesday September 23, 2009 @09:20PM
(#840445)
That and...
"Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed"
... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.