News for hackers. Stuff that matters.
Re:Ask YC: Where to start with creating a distributed crawler by konsl on Sunday March 30, 2008 @05:41PM (#150432)
If you're looking at building your own crawler in Python from scratch, here's a benchmark of SGML parsers:

http://72.14.205.104/search?q=cache:LYoRD1GTP2UJ:www.oluyede...

We've been playing with sgmlop (http://effbot.org/zone/sgmlop-index.htm) for parsing and urllib2 (http://docs.python.org/lib/module-urllib2.html) for fetching.

There are no comments on this comment yet.