Of the signals used to indicate crawling is prohibited, robots.txt is probably the most effective; OpenAI, Google, Anthropic, Meta, and CommonCrawl all claim to respect it. That often provokes a response of "well they're lying", but I've yet to actually find any cases of the IPs they use for crawling accessing content prohibited by robots.txt.

Newly proposed standards will probably take a while to catch on, if they ever do.

Not a lawyer, but I believe such measures could in theory become legally enforceable in the US without any new legislation if the fair use defense fails but an implied license defense (the reason you can cache/rehost copies of webpages that don't have a <noarchive> meta tag, as in Field v. Google Inc) succeeds.