News for hackers. Stuff that matters.
Re:A Curious Case of Disregarded Robots.txt by SyneRyder on Wednesday April 26, 2017 @12:00AM (#14200342)
That's a really good point. I'm not a fan of Internet Archive ignoring robots.txt, but if I'm really unhappy about it I can block their robot in Apache using .htaccess rules (as long as they continue using archive.org_bot as their user agent).

There are no comments on this comment yet.