Large language models trained on cross-entropy loss for next token prediction aren't going to be dangerous per se, even in the limit - simply due to the nature of the objective and the distribution they are set to approximate.
We should make sure these powerful tools don't fall into a small exclusive set of power-hungry interests, though. Transformative tools such as these should benefit all of humanity and let it prosper.
An AGI, by virtue of being at least as general as a human, would be able to effectively form a plan for how to act in the world to achieve fairly general goals, in fairly general conditions.
Even if it isn’t agentic, if a program can formulate these plans in a way people can use, then the combined program+user still implements the optimization process.
The democratic solution here looks better than the expected rule of the (very) few.