Trivially with robots.txt, which is standardized and so well known among anybody doing webservers that not having such a rule ought to be considered as consent.

About the same as the rule here in Sweden on filming - you're free to film in any private venue until told not to.