News for hackers. Stuff that matters.
Re:Python web scraping by Denzel on Tuesday March 11, 2014 @12:21PM (#7380243)
That's interesting. I'd love to hear more about your problems with it. My email is in my profile, feel free to drop me a line.

I deployed a small but moderately complex crawler backed by a database with Scrapy. It had some custom pipelining as well that deduplicated information (based upon a hash) and exponentially backed-off if the page hadn't changed in a while.

There are no comments on this comment yet.