News for hackers. Stuff that matters.
Re:The Internet Archive is waging war on misinformation by wahern on Thursday September 19, 2019 @02:10AM (#21013737)
I like the "different kings" metaphor. An archive has to think long-term. To do its job unimpeded it has to remain inconspicuous and unobjectionable, otherwise it may lose access to information altogether. Kings and cultures will come and go, but as long as they have the data then their mission is substantially fulfilled. At some point in the future even the most radical material will become innocuous and safe to publish.

For example, this is why the archive obeys robots.txt policies retrospectively. I can't access my personal web pages from 20+ years because at some point the university changed it's robots.txt policy. (The domain no longer exists, freezing that policy quasi-permanently.) But at least they're archived. Maybe at some point before I die I'll get to see them again, but at least it's a possibility. More importantly, my "sacrifice" (such as it is) is a small price to pay to help ensure that archive.org preserved access to material that would have been cut off if websites felt that they didn't have control of availability once archived.

We can't expect archive.org to fight our battles for us, whether regarding censorship, copyright overreach, or other issues. It's great that they do to the extent that they do, but their success is partly a consequence of them picking their fights and emphasizing their character of being a quiet, harmless archiver. It would be counter-productive to ask them to risk that credibility and power. And it's entirely unnecessary.

Re:The Internet Archive is waging war on misinformation by Roritharr on Thursday September 19, 2019 @03:21AM (#21014058)
My gut reaction was also:"oh no, not them, why don't they take a stand?" but your logic is completely valid, they shouldn't, for the futures sake. I just hope they ensure their source data stays untouched.

What surprised me was that you say they respect the robots.txt, but the data is archived anyway, so they are crawling anyway, but only publish according the rules defined in the robots.txt?

If so that is news to me.

Re:The Internet Archive is waging war on misinformation by Retric on Thursday September 19, 2019 @06:10AM (#21014849)
The robots.txt changed and they applied the new robots.txt to the old data.