Re:Scrapy Tips from the Pros
by mitchtbaum on Wednesday January 20, 2016 @07:20AM
(#10937549)
> 1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed?
Usually web clients use https://en.wikipedia.org/wiki/HTTP_ETag , afais. If a web app\server lacks that skill, then you could compute your own hash and check it yourself, instead of processing that condition at the network layer.
Usually web clients use https://en.wikipedia.org/wiki/HTTP_ETag , afais. If a web app\server lacks that skill, then you could compute your own hash and check it yourself, instead of processing that condition at the network layer.