News for hackers. Stuff that matters.
Re:A Curious Case of Disregarded Robots.txt by upofadown on Tuesday April 25, 2017 @08:22PM (#14199479)
A robots.txt file is not any sort of license (anti-license?). It's existence has no bearing on the question of if the IA should be allowed to do what it does. It is only intended to provide helpful information to web crawllers.

* http://www.robotstxt.org/norobots-rfc.txt

There are no comments on this comment yet.