Your example is an example of choice or consent. They also had the option to burn their hand written books and scrolls down periodically. Systems like IA take that choice away.
Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
You made it public on the damn web, people doing what they wanted with it is fair game per the original ethos of the web. If you wanted it to be private, you should have some auth or encryption.
The actual real world harm is in actually-common-used means of personal communication were network effects preclude on carrying out they business with technology appropriate to desired privacy levels. Chatting on my custom website was always niche, and therefore the network effects argument doesn't carry water.
We should refrain from worrying about robots.txt minute until the elephant in the room is put to rset.
Thanks.
Trying to muddle in privacy concerns with the accumulation of public knowledge undermines the whole concept of a shared society, or there being even the potential of accumulating "progress" in the first place.
Also, historically speaking, people with means used to save their letters for posterity, which proved to be a very valuable resource for future academics, so the idea of, what, deleting all your proton mails and signal messages as encouraged is arguably overshooting the return to some pre-internet norm.