1. The dangerous forms of AI may require very specific and unusual development paths, that could be sufficiently discouraged by collective action that they don't get pursued.
2. Even if all development of dangerous forms of AI cannot be stopped, attempts to minimize it may still have a meaningful impact because an arms race may develop between AI that counters dangerous AI, and dangerous AI, and roadblocks put in the way of the development of the latter could ensure the former stays ahead.
3. Dangerous forms of AI may require a significant foothold in the socio-economic space to become an existential threat, and concerted collective efforts to suppress it may prevent that critical threshold from ever being close to reached.
Perhaps doing "nothing" is not best. It may be wise to put restrictions on large, publicly available LLMs to minimize impact and for the practical purpose that it is the easy route for millions of people to use AI.
I think your points are valid.
I hope, though, that leaders and citizens are aware that trying to put a full lid on AI is like trying to outlaw math or censor information. It's like trying to wipe Wikipedia or ban encryption. Sure, you may try to take out a website or strongarm Apple/Meta into backdoors, but you can't stop two people from sending each other encrypted data.
Trying to govern the handful of companies with an edge right now, when this tech will be completely democratized in short time, is pointless.
Regulations and recommendations should center around the use of AI by individuals. Enforcing AI disclosure requirements for organizations that create lifelike images for parody, with some kind of watermark or overt disclosure.
Post a video of the president announcing a nuclear attack? Better have a clear "this is AI" message or it gets a takedown.