Why should we pay taxes, if the government won't even train AI for us, given such opportunity?
Large language models trained on cross-entropy loss for next token prediction aren't going to be dangerous per se, even in the limit - simply due to the nature of the objective and the distribution they are set to approximate.
We should make sure these powerful tools don't fall into a small exclusive set of power-hungry interests, though. Transformative tools such as these should benefit all of humanity and let it prosper.
An AGI, by virtue of being at least as general as a human, would be able to effectively form a plan for how to act in the world to achieve fairly general goals, in fairly general conditions.
Even if it isn’t agentic, if a program can formulate these plans in a way people can use, then the combined program+user still implements the optimization process.
The democratic solution here looks better than the expected rule of the (very) few.
And there we should just ask our new creative oracle machines the obvious: "devise an understandable minimally invasive method of uplifting our own cognition and extending our biological lifespan". Surely at least some gene/cell/whatever therapy which does what we need is theoretically possible - the human beings aren't special enough for this not to be the case.
At least that's the zeitgeist I perceive among others who understand the significance of the unfolding progress.