News for hackers. Stuff that matters.

A comment on Ask YC: Where to start with creating a distributed crawler.

Re:Ask YC: Where to start with creating a distributed crawler by iowahansen on Sunday March 30, 2008 @01:10PM (#150237)
1.) Don't do it yourself. Use Amazon's Alexa Web Search service (aws.amazon.com). Through that you can access Alexa's 10 billion page index, complete with all the pages, run complex queries etc. Plays nicely with EC2.

2.) If you must do it yourself, Heritrix is the most sophisticated crawler out there (crawler.archive.org).

3.) Nutch is an option, but nowhere near as powerful as Heritrix.

Don't try to reinvent the wheel, writing a robust crawler is a lot of work as there are endless edge cases to take care of (if you are looking into a general purpose web crawler)

[ Parent | on HN ]
Re:Ask YC: Where to start with creating a distributed crawler by sonink on Sunday March 30, 2008 @01:38PM (#150272)
Nutch is good and I would second it, but I would suggest to NOT build a crawler - its not trivial and inadvised in a startup, that is unless your startup is just about building a crawler.
[ Parent | on HN ]