News for hackers. Stuff that matters.

A comment on Ask YC: Any ideas about intelligent crawlers :).

Re:Ask YC: Any ideas about intelligent crawlers :) by mig on Tuesday January 08, 2008 @02:45PM (#96072)
Don't write your own crawler. Use nutch.

It is designed to scale and do mapreduce kind of parallel processing. I would strongly recommend you to take a look before writing your own.

http://lucene.apache.org/nutch/

[ Parent | on HN ]
Re:Ask YC: Any ideas about intelligent crawlers :) by imsteve on Wednesday January 09, 2008 @09:03AM (#96337)
mapreduce? Just how many requests will you be making to third-party sites at once? Sounds like a good way to get blocked fast.
[ Parent | on HN ]