Re:A Curious Case of Disregarded Robots.txt
by SyneRyder on Wednesday April 26, 2017 @12:00AM
(#14200342)
That's a really good point. I'm not a fan of Internet Archive ignoring robots.txt, but if I'm really unhappy about it I can block their robot in Apache using .htaccess rules (as long as they continue using archive.org_bot as their user agent).
[ Parent
| on HN ]
There are no comments on this comment yet.