Scrapy can do the job, for sure. We use it to crawl more than 2 billion pages a month. :)