Is failure to honor a robots.txt a crime? Or rather, would it be unlawful to spoof a user agent to access this publicly available data? After the linkedin [0] case it seems reasonable to think not.

[0]: https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

Spoofing user-agents hasn't worked in a long time for anything but small operations because search engines publish specific IP ranges their scrapers use.

The CFAA is so broad and broadly interpreted that I would assume that failure to honor any any site's robots.txt file may incur criminal liability if the U.S. government can claim American jurisdiction (e.g., because the site's owners are U.S. persons or a U.S. corporation, or because the site's servers are located in the U.S.).

> Is it legal for a government entity to issue a robots.txt like that?

I may be wrong (this isn't my area), but I was under the impression that robots.txt was just an unofficial convention? I'm not saying people should ignore robots.txt, but are there legal ramifications if ignored? I'm not asking about techniques sites use to discourage crawlers/scrapers, I'm specifically wondering if robots.txt has any legal weight.

Yep, it's defined at https://www.robotstxt.org/

Looks like Google is trying to turn it into an RFC though.