Scrapy is great. I have been using it at work to populate our solr index for our site search page. It is very extensible and powerful, but helps to understand Twisted if you need to do anything advanced.