News for hackers. Stuff that matters.
Re:An Introduction to Compassionate Screen Scraping by megamark16 on Tuesday April 19, 2011 @03:09PM (#2463804)
It was my understanding the urllib2 respects robots.txt automatically. I can't find much to back that up, but I really thought I read that somewhere reliable once.
Re:An Introduction to Compassionate Screen Scraping by mcav on Tuesday April 19, 2011 @03:41PM (#2463920)
This link corroborates that urllib2 respects robot.txt:

http://stackoverflow.com/questions/3197299/urllib2-connectio...

Re:An Introduction to Compassionate Screen Scraping by arst on Tuesday April 19, 2011 @05:32PM (#2464317)
That link is wrong. Python does ship with a robotparser module in the standard library that parses robots.txt files, but urllib2 does not use it out of the box. This can be easily confirmed using Wireshark or a quick glance at the source: http://hg.python.org/cpython/file/08b5e2c9112c/Lib/urllib2.p....