Re:Scrapy Tips from the Pros
by binarysolo on Wednesday January 20, 2016 @08:30PM
(#10942775)
Seconding the practice of collect, then parse.
Storage is cheap relative to hammering a server (or working around the bounds of rate-limitation), so always save your collected things, then crawl it as many time as you desire as you play around with the right scrapes.
Storage is cheap relative to hammering a server (or working around the bounds of rate-limitation), so always save your collected things, then crawl it as many time as you desire as you play around with the right scrapes.