Further, setting up a physical device connected to a public IP is never default behavior so you are putting up the sign in either case. So, at best your argument is someone athorised to do something put up a sign by mistake saying something that was not intended, but your intent has little relevance at that point.
Worse, your argument is based on the assumption that nobody knew what was going on so even simple coursework mentioning robots.txt would demonstrate knowledge and thus intent through willful inaction.
There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?
Not really. There are often default robots.txt files that the system just puts there in the course of building a default website.
The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.