News for hackers. Stuff that matters.
Re:Ask YC: Existing Crawlers in Python by yourabi on Tuesday September 09, 2008 @10:42AM (#299434)
I would also take a look at Heritrix (http://crawler.archive.org/) -- it's what powers the wayback machine.
Re:Ask YC: Existing Crawlers in Python by gojomo on Tuesday September 09, 2008 @01:32PM (#299612)
Thanks for the plug!

As a developer of Heritrix, I can't honestly say it's compact or Python, but it is well-behaved, highly customizable (both by settings and by many Java extension points), and capable of high-volume crawling for many purposes.

You could also embed Python code via Jython with a little work, if necessary.