Re:Ask HN: Is there a great open source crawler?
by Ledio on Wednesday June 02, 2010 @11:19PM
(#1400294)
Nutch has a full on web crawler, a lot of features, and it scales pretty well. You can white list or black list URLs as you see fit, and filter out unwanted content.
[ Parent
| on HN ]
There are no comments on this comment yet.