Re:Crawling the web at $2 per million pages
by jdrock on Thursday September 24, 2009 @01:27AM
(#840660)
Just a quick heads up - we'll be looking into allowing JRuby and Jython fairly soon for when you create 80Apps.
You'll also be able to mashup third-party apps and libraries into your own code. This is something we hope to have in 2-3 months.
Re:Crawling the web at $2 per million pages
by coderdude on Thursday September 24, 2009 @05:10PM
(#842206)
How come there isn't a REST API? Java, JRuby, Jython. Ewwww. This isn't how outfits are doing things anymore...
Re:Crawling the web at $2 per million pages
by jdrock on Sunday September 27, 2009 @11:32AM
(#846918)
The crawl runs on a super-heterogenous network of computers from around the world. The JVM sandbox is the only thing we know. Java is the only language that has something like this.
Yes, Java isn't as sexy anymore. But running your code on 50,000 computers sounds sexier to me than a REST API.
However, if you want to run crawling on your own infrastructure, I recommend Scrapy (http://scrapy.org/), a python crawling framework introduced on HN last year. Scrapy solves some of the more time-consuming problems involved in writing a crawler from scratch (multiple simultaneous requests, pipelined processing, raw caching, duplicate URL filtering) and comes with nifty development and administration tools. More importantly, it has an active and helpful set of core developers and good documentation. I am comfortable with both Python and Java but I chose Scrapy over 80legs because I can crawl for free on the machines I already have and I can afford to spend more time crawling from a single IP compared to 80legs which will let me crawl much faster but isn't free. Also, with Scrapy my bot can be 'naughty' - 80legs jobs obey robots.txt and limit the crawling rate per domain.
When my crawling needs outgrow my infrastructure, I am going to look at 80legs again.