Re:Scrapy Tips from the Pros
by rcfox on Wednesday January 20, 2016 @05:19PM
(#10941820)
The very first thing I do with every scraping project is enable the HttpCacheMiddleware[0]. After downloading a page once, subsequent runs will automatically pull it from the local cache. This makes it way faster to experiment, and doesn't increase the burden of the website.
[0] http://doc.scrapy.org/en/latest/topics/downloader-middleware...