Archiveteam projects are different because we run closely-monitored, highly targeted crawls. If a site has an infinite hallway in it, we'll notice that and exclude it while taking steps to replicate enough of it to retain its valuable attributes. We also do our best to avoid retrieving content more times than necessary. If you operate a database, we try to get enough pages to replicate a good portion of the underlying data.
Maybe, but does that make robots.txt useless? I don't think so.
Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing?
The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves?
If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.
As the "some crazy reason" might be that the database contains data that is not publicly addressable for a reason and sanitising the data-set could be non-zero effort operation, letting you crawl the public pages and do some work at your side would be the sensible route from most points of view. Heck, even if there is definitely not anything not already displayed to the public in the DB, letting you go ahead and get the information the hard way is likely to be preferable to lifting a finger to send you a DB dump.
Sort of like a cat's attitude to playing fetch: You want it? You go get it. I've got more important things to do.
I think they're saying is that their archive is going to ignore it, and your website should just eliminate it.
We'll have to assume that archiveteam.org are by now real experts in guessing which buttons are going to be pissing people off. :-)
The daftness: maybe the claim is true that robots.txt was only a stop-gap measure back when web servers sucked, however the de facto modern use for it goes far beyond that, and ignoring that standard is likely to piss off lots of people.
The insight: for crawlers, relying on robots.txt to prevent getting stuck indexing infinite hierarchies of data is a bad idea. It should be able to figure that much out for itself, so it doesn't explode when faced with sites that don't exclude such hierarchies using robots.txt.
For servers, relying on a client hint to ensure reliability is daft. It should have some form of rate limiting built in, as that's the only sensible design. This seems the only marginally sensible use of robots.txt from a server standpoint. Using it for any form of security (e.g. preventing DB scraping) is daft, and a more robust mechanism should be employed there too.