News for hackers. Stuff that matters.
Re:Robots.txt as a Security Measure? by hjalle on Wednesday July 10, 2019 @06:06AM (#20400566)
Funny experiment and perhaps also useful, but there are crawlers with good intentions[1] that still may ignore the disallows. I don't know of anyone else than the internet archive though.

[1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

Re:Robots.txt as a Security Measure? by zaarn on Wednesday July 10, 2019 @09:02AM (#20401536)
Those crawlers can almost always be recognized by the UA.
Re:Robots.txt as a Security Measure? by hjalle on Wednesday July 10, 2019 @09:08AM (#20401573)
Yes, no doubt.