Not just the site owners either. I'm guessing not many people on here saw the early days of Gamergate, but one of the things that stoked the fire was the Internet Archive kindly erasing their archives of other websites' pages discussing the mess on request from a person being discussed after their hosting providers were leaned on to give them the boot, changing the past to fit the correct official narrative. It's one of the reasons why no-one involved with that uses them for archiving anything. (Wasn't doxing or anything like that, just explaining what happened was enough.)

In our modern hyperpoliticized era, one has to wonder if the Internet Archive is actually trustworthy for anything contentious anymore.

Adding a robots.txt file to a domain doesn't cause them to delete their archive of it, only to hide it.

Which is one and the same to the public.

IMO, they need to stop applying robots.txt retroactively if they want to be considered a valid archive.

The problem is that the Internet Archive exists on legally shaky ground. Neither they nor anyone else has a right to archive copyrighted web content and display it to the public. They manage to continue doing so in part because they're clearly non-commercial. They also manage to continue doing so because they voluntarily respond to robots.txt, even retroactively.

Libraries/archives have no special exemption from copyright law, which is actually a good thing, because otherwise libraries would presumably need to be licensed in some way by the government to qualify for special treatment.

Why not look at WHOIS information when getting an update, and then class a site as 'different' based on whether that changes? In most cases, a new domain owner usually means the site isn't the same as the earlier versions.

You'd then just have to stop the archive indexing/showing content after the WHOIS information changed, while leaving the stuff before it intact. Maybe you'd then have a nice form to report pages you want removed/hidden (for the edge cases), or even a seperate robots.txt/meta declaration you can make confirming you're the same person that owns the site. After all, most of the reasons why sites go missing aren't deliberate attempts to rewrite history, but domain squatters not wanting holding pages indexed.

Feels like it'd be so easy to implement robots.txt in a more logical way on the Internet Archive.

It's been suggested, but there's no way to automatically do it correctly. The whois info might be anonymized, in which case a change means nothing at all. It might just be someone's name and address, with no way of verifying who that someone works for. Also meaningless. Better just to default to something safe, and spend your manpower on something more important.

    > the Internet Archive exists on legally shaky ground

Not least because of the EU's 'right to be forgotten'.

That doesn't seem likely to apply.

Key point given the current climate, if the Trump presidency adds a restrictive robots.txt to all .gov domains they will prevent the Internet Archive Wayback Machine from showing history on any of those domains.

Not only would this obliterate public access to the Obama, Bush, Clinton era government websites in the archive, it'd prevent the use of the Wayback Machine for keeping track of Trump's shifting agendas, as demonstrated on his web domain recently.

Government documents including, I assume, web pages are in the public domain.

That's irrelevant; the issue isn't copyright, but the implementation of the Internet Archive, which applies robots.txt retroactively to archived versions of pages as well as respecting it currently.

But the reason the Internet Archive applies robots.txt retroactively is copyright, as explained in a sibling to the grandparent.

So someone* should make a clone of archive.org but built on Tor. Maybe call it Torchive. And with no hiding because of robots.txt and possibly prevent users from looking at/editing the data on their server so a person who wants to clear themselves can't flood the net with nodes wanting data on this person and then wipe them all.

*someone who has the skills to make it, because I don't, at least right now

Ok, but, I'm very thankful I was able to remove the site I made 20 years ago when I was 14.