Re:Ask YC: Any ideas about intelligent crawlers :)
by imsteve on Wednesday January 09, 2008 @09:02AM
(#96336)
It's called "scraping" and I've done it lots of times with python, very easy. Don't bother with those other specialized, non-python, frameworks that people are suggesting.
http://wwwsearch.sourceforge.net/mechanize/
And if you need to do complicated html parsing in combination with that:
http://www.crummy.com/software/BeautifulSoup/
From there, it's cake.