Re:An Introduction to Compassionate Screen Scraping
by arst on Tuesday April 19, 2011 @05:32PM
(#2464317)
That link is wrong. Python does ship with a robotparser module in the standard library that parses robots.txt files, but urllib2 does not use it out of the box. This can be easily confirmed using Wireshark or a quick glance at the source: http://hg.python.org/cpython/file/08b5e2c9112c/Lib/urllib2.p....
[ Parent
| on HN ]
There are no comments on this comment yet.