The civics surrounding the rights and responsibilities of website and server operators really should be communicated better; a more informed general public would allow better regulation and legislation to be created.
Robots.txt should have the legal standing of a No Trespassing sign. You should be able to dictate terms to a reasonable degree on a public site; it shouldn't be legal to effectively DOS someone endlessly and repeatedly without recourse. As it stands, though, there's not a lot of protections for little guys. You have to implement a login mechanism, or some way of identifying unique visitors and requiring authentication, in order to impose your own rules on a site.
If I publish content at my domain, I can set up blocklists to refuse access to IP ranges I consider more likely to be malicious than not. Is that not already breaking the social contract you're pointing to wrt serving content public ? picking and choosing which parts of the public will get a response from my server ? (I would also be interested to know if there is actual law vs social contracts around behavior) So why shouldn't I be able enforce expectations on how my server is used? The vigilantism aspect of harming the person breaking the rules is another matter, I'm on the fence.
Consider the standard warning posted to most government sites, which is more or less a "no trespassing sign" [0] informing anyone accessing the system what their expectations should be and what counts as authorized use. I suppose it's not a legally binding contract to say "you agree to these terms by requesting this url" but I'm pretty sure convictions have happened with hackers who did not have a contract with the service provider.