News for hackers. Stuff that matters.
Re:Ask HN: Is there a great open source crawler? by ericwaller on Wednesday June 02, 2010 @11:22PM (#1400297)
Scrapy (http://scrapy.org/) matches your description pretty well. You can specify which urls to crawl with regular expressions and then provide a bit of code to do some data extraction.

There are no comments on this comment yet.