News for hackers. Stuff that matters.
Re:Crawling the web at $2 per million pages by kbrower on Wednesday September 23, 2009 @04:47PM (#840091)
I would love it if it processed, stored, and indexed the data.

http://parselets.com/ + mysql + sphinx

Re:Crawling the web at $2 per million pages by qeorge on Wednesday September 23, 2009 @05:46PM (#840185)
Wow - parselets.com is amazing. Thanks for sharing.

I think 80legs can do some level of parsing now, but Parsely seems like a great way to describe the analysis for targeted crawls on predictable data. If it supported the combination you've described it would really open up some cool possibilities.

Re:Crawling the web at $2 per million pages by jdrock on Thursday September 24, 2009 @01:34AM (#840665)
We are looking at storing data, doing pre-done crawls, offering data feeds, etc. It's something on the horizon for us.

Hopefully guys that write parselets will be interested in becoming developers for the upcoming 80legs App Store. They'll be able to sell their parselets and earn 100% recurring revenue from 80legs users using their parselets.

Re:Crawling the web at $2 per million pages by lsb on Wednesday September 23, 2009 @04:59PM (#840109)
Try Yahoo Pipes for things like that?