News for hackers. Stuff that matters.
Re:Ask HN: MP3 Crawler by deutronium on Tuesday July 14, 2009 @02:13PM (#704249)
You may be interested in Nutch (http://lucene.apache.org/nutch/), an Open Source crawler, which handles indexing etc. for you. Its also based on Hadoop, so it should scale nicely just by throwing more machines at the job.

There are no comments on this comment yet.