News for hackers. Stuff that matters.
Re:Crawling the web at $2 per million pages by tdavis on Wednesday September 23, 2009 @05:05PM (#840123)
You'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probably take a few more to make it "fill in a web form and go" usable). After that, the processing is pretty straight forward.

You've got a URL, headers, and body content. Just extract what you want and crunch.

There are no comments on this comment yet.