News for hackers. Stuff that matters.
Re:Ask HN: Is there a great open source crawler? by Ledio on Wednesday June 02, 2010 @11:19PM (#1400294)
Nutch has a full on web crawler, a lot of features, and it scales pretty well. You can white list or black list URLs as you see fit, and filter out unwanted content.

There are no comments on this comment yet.