Re:Python web scraping
by Denzel on Tuesday March 11, 2014 @12:21PM
(#7380243)
That's interesting. I'd love to hear more about your problems with it. My email is in my profile, feel free to drop me a line.
I deployed a small but moderately complex crawler backed by a database with Scrapy. It had some custom pipelining as well that deduplicated information (based upon a hash) and exponentially backed-off if the page hadn't changed in a while.
I deployed a small but moderately complex crawler backed by a database with Scrapy. It had some custom pipelining as well that deduplicated information (based upon a hash) and exponentially backed-off if the page hadn't changed in a while.