I have always wondered, is there legal precedent here? If someone were to ssh into your server without permission that's pretty obviously illegal access. But the web is just another protocol on another port, what makes it different?

I certainly never said Google could crawl my website, and I would be surprised if the presence of a robots.txt file is written into law as a requirement to prove you didn't want them on your web properties.

I feel like this must have been hashed out in lawsuits already, probably against search crawlers.

I'm too tired for a thorough search, but here's an article which states that an appeals court decided it's legal: https://techcrunch.com/2022/04/18/web-scraping-legal-court/

IANAL and so on.

i m sure a lot of people lobbied for this to not be legislated

And I'm pretty sure you'd be wrong. For example, uploading a movie you don't have a copyright to to YouTube is illegal. That movie night be public data, but you don't have the permission of the copyright holder to copy that data.

Publicly accessible =/= You having the right to do whatever you want with something.

A French famous white hacker got condemned in French for publishing confidential information found through a Google search. The key thing in the case was that he undeniably aware that this was just a file server misconfiguration and that he wasn't supposed to be able to access it.

If you want more info about the case you should look for the "affaire Bluetouff*.

Leaving your car door open isn't a great analogy.

The internet is closer to leaving your own journal open at the library or a magazine store, and people going through it. You made it available for everyone to peek at in a public space. Maybe you didn't intent to leave it there, maybe you didn't intend to make it visible, but you did.

That is a fair analogy but still, me leaving it there doesn’t grant you the right to copy down what’s written in it and use it for something else.

your car can be parked in a public street

Indeed. Microsoft’s name and logo is published everywhere “in public” but you certainly can’t use them as your own.

IP is still IP even when it’s publicly available.