News for hackers. Stuff that matters.
Re:The Internet Archive is waging war on misinformation by Roritharr on Thursday September 19, 2019 @03:21AM (#21014058)
My gut reaction was also:"oh no, not them, why don't they take a stand?" but your logic is completely valid, they shouldn't, for the futures sake. I just hope they ensure their source data stays untouched.

What surprised me was that you say they respect the robots.txt, but the data is archived anyway, so they are crawling anyway, but only publish according the rules defined in the robots.txt?

If so that is news to me.

Re:The Internet Archive is waging war on misinformation by Retric on Thursday September 19, 2019 @06:10AM (#21014849)
The robots.txt changed and they applied the new robots.txt to the old data.