News for hackers. Stuff that matters.
Re:Scrapy Tips from the Pros by rcfox on Wednesday January 20, 2016 @05:19PM (#10941820)
The very first thing I do with every scraping project is enable the HttpCacheMiddleware[0]. After downloading a page once, subsequent runs will automatically pull it from the local cache. This makes it way faster to experiment, and doesn't increase the burden of the website.

[0] http://doc.scrapy.org/en/latest/topics/downloader-middleware...

There are no comments on this comment yet.