Re:A Curious Case of Disregarded Robots.txt
by true_religion on Tuesday April 25, 2017 @08:31PM
(#14199526)
Well yes, the whole point of robots.txt is that it's impolite to refuse to follow it, and doing so and getting caught might get you banned from the site.
[ Parent
| on HN ]
Re:A Curious Case of Disregarded Robots.txt
by SyneRyder on Wednesday April 26, 2017 @12:00AM
(#14200342)
That's a really good point. I'm not a fan of Internet Archive ignoring robots.txt, but if I'm really unhappy about it I can block their robot in Apache using .htaccess rules (as long as they continue using archive.org_bot as their user agent).