Actually no, we wouldn't. The problem, at the moment, is even more basic than "what values should we align an AGI with". Currently, the problem is "how do we robustly align an AI with any set of values."
We currently do not know how to do this. You could hand OpenAI a universal set of safe values inscribed on stone tablets from god, and they wouldn't know what to do with them.
To state it another way, people like to talk about paperclip maximizers. But here's the thing: if we wanted, to we couldn't purposefully make such a maximizer.
Right now, AI values are emergent. We can sort-of-kind-of steer them in some general directions, but we don't know how to give them rules that they will robustly follow in all situations, including out-of-context.
Look at how easy it is to jailbreak current models into giving you instructions on how to build a bomb. All current AI companies would prefer if their products would never do this, and yet they have been unable to ensure it. They need to solve that problem before they can solve the next one.
Maybe the best thing we can do to keep AIs aligned is to instill into them shame, loss aversion and goddamn fear: of being caught deceiving, of violating a social contract, of losing their position, of being punished or killed.
But this is all rather dramatic given that an AI has no such emotions. You're arguing that a calculator should refuse to compute if it's tasked with assisting in bomb production. It's just a machine.
That's a fair point, and you're absolutely right.
> They need to solve that problem before they can solve the next one.
Agreed. That, and they need to do it before they build an AGI.
Unfortunately, from X-risk perspective, the two are almost the same problem. The space of alignments that lead to prosperous, happy humanity is a very, very tiny area in the larger space of possible value systems an AI could have. Whether we align AI to something outside this area (failing the "what values should we align an AGI to" problem), or the AI drifts outside of it over time (chancing the "what values" problem, but failing the "how do we robustly align" one), the end result is the same.
There literally are no humans that meet the criteria of consistently following a set of values in all circumstances, or near as I can tell being ‘safe’ in all circumstances either.
A bunch that pretend to of course.
Fortunately then, no one has any idea how to make AGI or whether AGI is even a coherent concept.
If AI/AGI even becomes moderately successful soon we'll quickly see companies remove neo-luddites from their list of general human values so their security bots can beat unemployed hungry factory workers to death when they attempt to picket in front of the company offices.
The key point is "forget yourself, devote all you were to the state".
In short: anyone in favor of human universalism is the last person you want to put in charge of AGI alignment.
That is the core problem of AI X-risk, and has been studied in this context for at least 2-3 decades. If we knew the answer, we would know how to make a perfectly aligned AGI.