Re:Robots.txt as a Security Measure?
by hjalle on Wednesday July 10, 2019 @06:06AM
(#20400566)
Funny experiment and perhaps also useful, but there are crawlers with good intentions[1] that still may ignore the disallows. I don't know of anyone else than the internet archive though.
[1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...