News for hackers. Stuff that matters.
Re:Ask YC: Any ideas about intelligent crawlers :) by imsteve on Wednesday January 09, 2008 @09:02AM (#96336)
It's called "scraping" and I've done it lots of times with python, very easy. Don't bother with those other specialized, non-python, frameworks that people are suggesting.

http://wwwsearch.sourceforge.net/mechanize/

And if you need to do complicated html parsing in combination with that:

http://www.crummy.com/software/BeautifulSoup/

From there, it's cake.

There are no comments on this comment yet.