No, put up a loginwall or paywall, authenticate users, and go private. You can't have the benefits of a public site and try to impose your own terms and conditions. This is why LinkedIn had to go private and require logins; the public internet operates by technical protocols, and by serving content, you expressly grant permission to receive the content, without restriction or qualification. It's a proactive transaction on the part of the server operator. If you want to serve to any old person or system or software on the internet, and your server is configured to do so, you cannot impose arbitrary additional rules after the fact. You expressly consent to the user's interactions with your server by setting it up to serve content to whoever connects.

If you want ToS or rules of conduct or any other constraints on interactions with your server or content, you have to engage the user in a legally binding contract. The technical mechanisms to do so require authenticating the user, requiring identification and segregation of individuals, each requiring separate, documented consent and agreement. Once there's a login, or unique session, you can impose whatever conditions you want, and you should.

Interacting in public comes with certain expectations and limitations. This should be communicated more clearly; the general public should know what it means to publish to a platform open to the public, for example.

> No, put up a loginwall or paywall, authenticate users, and go private.

We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they will jump through hoops to take it anyway.

https://www.theguardian.com/technology/2025/jan/10/mark-zuck...

If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.

They have to follow the law, and the law explicitly allows scraping and crawling. Norms aren't enough; to get where most people seem to want to be, we need legislation that effectively makes robots.txt as legally binding and valid as a No Trespassing sign. There are significant consequences to the internet and public discourse if that happens, and it's not all good.

this is a very US-ian view of the world

my site is not in the US, I am not a US citizen. US law does not apply to me.

under UK law: robots.txt is an access control mechanism (weak or otherwise)

knowingly bypassing it is likely a criminal offence under the Computer Misuse Act

good luck suing me because you got stuck when you smashed my window and climbed through it

>the law explicitly allows scraping and crawling.

Nepenthes also allows scraping and crawling, for as long as you like.