What the archive can do after that point is a different issue, but they clearly can keep a copy. Further, someone else is using the domain they don't nessisarily have anything to do with the archived data.
Google and others have enhanced robots.txt to enable permission for crawling (allow, sitemap), meta tags can deny archiving and various means allow permission to be explicitly denied for caching.
To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal.
FWIW I disprove of this state of affairs and consider copyright to be hugely defective in these respects.
>but they clearly can keep a copy //
It's nuanced but permission to access a page =/= permission to keep a copy. Just as you have explicit permission to access a video on YouTube but in most jurisdictions will not have permission to download it for later (commercial) use.
Right. And it's actually not a bad analogy as analogies go. Not having a sign doesn't make trespassing legal but if someone sometimes walks over a corner of your property and you go to the police to try to get him arrested, the first thing they'll probably ask you is if you have your property posted and/or if you bothered to ask him to stop. If the answer is no, they'll probably tell you to go away and do so and only come back if he ignores the sign.
The requirement to post a sign to make trespassing an actionable offence is a USA thing AIUI, it's not a UK thing at least, but copyright is almost universal and doesn't require even adding a (c) mark, it's automatic at the point of creation under the Berne Convention. Or in other words you've pushed your analogy too far and hit a marked legal difference between USA physical property law enforcement and international intellectual property law.
The more general principle is that if no harm is done and the individual/organization will just stop the action if you asked, the courts are often reluctant to get involved. There are exceptions of course, especially in the vein of making an example of someone to discourage others.
Also, it's meaningless for a bot to get permission without having permission to make a copy. There are arguments around the number of copies, but the clear implication is at lest all the routers can make a copy.
Not really. There are often default robots.txt files that the system just puts there in the course of building a default website.
The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.
Further, setting up a physical device connected to a public IP is never default behavior so you are putting up the sign in either case. So, at best your argument is someone athorised to do something put up a sign by mistake saying something that was not intended, but your intent has little relevance at that point.
Worse, your argument is based on the assumption that nobody knew what was going on so even simple coursework mentioning robots.txt would demonstrate knowledge and thus intent through willful inaction.
There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?
But they're not a target that you're going to collect big from, as a non-profit archive/library they're sympathetic whether or not that gives them any special legal standing, and they'll basically take down your content past and present if you ask them to.
So it will almost certainly cost you money to sue them, you won't collect much in the best case, and you can get your content taken down in about as much time as it would take you to pick a lawyer out of the phone book.
archive.org fucked up by making robots retroactive, if they used the archived robots.txt as a filter for a site at the relevant date, they'd have had the best of both worlds - respecting how sites work without losing how sites appeared at a date.