News for hackers. Stuff that matters.
Re:Crawling the web at $2 per million pages by sachinag on Wednesday September 23, 2009 @04:42PM (#840081)
You know, I'd try this, but the hard part is that the processing is still on me (and I can't code, so I'm SOL). That, frankly, is the hard part, is it not?
Re:Crawling the web at $2 per million pages by jennwilson on Wednesday September 23, 2009 @06:01PM (#840211)
Hi, I work for 80legs and wanted to address this--we will be building an "App Store" over the next couple of months that will allow you to use Apps written by other developers to do some pretty cool stuff. We already have several of these 80Apps (as we call them) available, many of which have been written by the semantic search engine company swingly.com. You can view descriptions for them on the "Create Job" page if you sign up to use our Portal.
Re:Crawling the web at $2 per million pages by tdavis on Wednesday September 23, 2009 @05:05PM (#840123)
You'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probably take a few more to make it "fill in a web form and go" usable). After that, the processing is pretty straight forward.

You've got a URL, headers, and body content. Just extract what you want and crunch.

Re:Crawling the web at $2 per million pages by jdrock on Thursday September 24, 2009 @01:29AM (#840662)
While complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need several servers, IP addresses, and much more.