Restricting access to nuclear weapons is feasible, because of how hard they are to make on a larger scale, and even then it's very difficult. They are significant projects for entire nation states to undertake.

Training a LLM is something random companies with some cash and servers are doing.

So the outcome of "pretty much only we have nukes" is an option. I don't think that's a realistic proposal for AI. Given that, is the outcome of "more trustworthy people stop, others continue" a good one?