News for hackers. Stuff that matters.
Re:A Curious Case of Disregarded Robots.txt by true_religion on Tuesday April 25, 2017 @08:31PM (#14199526)
Well yes, the whole point of robots.txt is that it's impolite to refuse to follow it, and doing so and getting caught might get you banned from the site.
Re:A Curious Case of Disregarded Robots.txt by SyneRyder on Wednesday April 26, 2017 @12:00AM (#14200342)
That's a really good point. I'm not a fan of Internet Archive ignoring robots.txt, but if I'm really unhappy about it I can block their robot in Apache using .htaccess rules (as long as they continue using archive.org_bot as their user agent).