Re:An Introduction to Compassionate Screen Scraping
by hung on Tuesday April 19, 2011 @02:38PM
(#2463678)
Caching is nice, but HTTP has a built-in method: conditional GETs. I wrote up a blog post on how to do this with App Engine but it should work generally in Python using urllib2.
http://www.hung-truong.com/blog/2010/12/01/conditional-gets-...