News for hackers. Stuff that matters.
Re:The Internet Archive is waging war on misinformation by dredmorbius on Wednesday September 18, 2019 @05:43PM (#21010568)
IA are specifically refusing to follow robots.txt directives largely on account of just this behaviour.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

https://archive.org/post/184024/robotstxt-policy-is-a-failur...

Re:The Internet Archive is waging war on misinformation by Rebelgecko on Wednesday September 18, 2019 @06:23PM (#21010998)
That's great news. Is there any update to the 2017 blog post? Is the new policy still only for .mil and .gov sites?
Re:The Internet Archive is waging war on misinformation by dredmorbius on Wednesday September 18, 2019 @06:39PM (#21011186)
I don't know, though you could research via their blogsite.