News for hackers. Stuff that matters.

A comment on Ask YC: Where to start with creating a distributed crawler.

Re:Ask YC: Where to start with creating a distributed crawler by wehriam on Sunday March 30, 2008 @01:01PM (#150229)
I have recently written a Beautiful Soup / Twisted crawler. To make it distributed, presumably we'd use Amazon's queue service.

Feel free to get in touch if you're interested in the details.

[ Parent | on HN ]
Re:Ask YC: Where to start with creating a distributed crawler by groovyone on Sunday March 30, 2008 @04:34PM (#150398)
Thanks John. I'll do that. Your resume looks awesome mind. Not sure how much we'd be able to help you!
[ Parent | on HN ]