Re:A Curious Case of Disregarded Robots.txt
by upofadown on Tuesday April 25, 2017 @08:22PM
(#14199479)
A robots.txt file is not any sort of license (anti-license?). It's existence has no bearing on the question of if the IA should be allowed to do what it does. It is only intended to provide helpful information to web crawllers.
* http://www.robotstxt.org/norobots-rfc.txt