>Do people feel this way about the FDA, the FAA, etc?

To a degree sometimes. There is no benefit to regulators to allow for more risk, because they will be blamed if something goes wrong. There's hardly anything to gain (as a regulator) to take risk.

Completely agree, but that’s not the same as saying that these organizations do not provide benefit

Yes, as a matter of fact, I do. If you look at even the US regulatory environment, for example: FDA approval for new drugs takes decades and billions, and enough said on America's "War on Drugs". The NRC has practically strangled nuclear power in its cradle, and environmental review adds years and millions to practically all new infrastructure. Health insurance regulation and financial regulation have enshrined a few oligopolistic players, with nearly impossible regulatory burdens for any new entrants. It's not merely that any one regulatory agency has run amok. All such agencies are fundamentally incentivized to increase their power and reduce risk by adding regulation in a one-way ratchet, while they have very little incentive to allow innovation except sustained external pressure, which new industries rarely can sustain. Large corporations rarely counterbalance this regulatory creep: they welcome regulation, since they can capture agencies to their will through lobbying and the "revolving-door" and afford a legal staff, while startups and individuals still face impossible regulatory burdens. Don't even get started on copyright. Regulation in America is fundamentally broken, and while it admittedly does manage to "protect" consumers from some of the worst potential abuses, it does so by choking innovation and essentially freezing us in the 1970s.

Why should we believe that AI regulation would be any different? Whatever agency is given power won't just stop at reasonable guidelines. They will likely be pressured by big players like Microsoft to choke off open source, by copyright giants like the RIAA and Disney to stop generation, and by every imaginable constituency to protect their jobs from change. Most importantly, individual open-source development will become prohibitive, and AI will be locked behind corporate APIs. In fact, if you look at sites like LessWrong, AI doomers are openly welcoming regulation precisely because they know an FDA-like agency will stop progress dead in its tracks. Make no mistake, AI regulation will hand enormous power to governments and corporations while denying it to the individual, whatever the good intentions were at the beginning. It is a devil's bargain, and if we as an industry are to take it in the name of "safety", we should at least do it with eyes open, instead of pretending that all that is being asked for is "guardrails" and "common sense".

Maybe we disagree on a fundamental level then. Even though the FDA, NRC, FAA, etc overregulate and stifle somewhat, I think they are a net good and that a regulation-free environment would not be a better situation.