Re:The Internet Archive is waging war on misinformation
by Rebelgecko on Wednesday September 18, 2019 @06:23PM
(#21010998)
That's great news. Is there any update to the 2017 blog post? Is the new policy still only for .mil and .gov sites?
Re:The Internet Archive is waging war on misinformation
by dredmorbius on Wednesday September 18, 2019 @06:39PM
(#21011186)
I don't know, though you could research via their blogsite.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
https://archive.org/post/184024/robotstxt-policy-is-a-failur...