Re:Analyzing One Million Robots.txt Files
by CM30 on Monday December 25, 2017 @08:51AM
(#16004077)
Honestly, I'm kind of surprised that turnitin's bot listens to robot.txt, or that the 'anti copyright infringement' bots do the same. Seems like it provides a very simple way for a cheating site to just thwart their entire 'system'.
But hey, I guess it's one of those cases where the law and basic ethics clash a bit; with certain laws saying 'unauthorised' access to a server is illegal, then ignoring that would leave them under fire for that instead.
But hey, I guess it's one of those cases where the law and basic ethics clash a bit; with certain laws saying 'unauthorised' access to a server is illegal, then ignoring that would leave them under fire for that instead.