Maybe the best thing we can do to keep AIs aligned is to instill into them shame, loss aversion and goddamn fear: of being caught deceiving, of violating a social contract, of losing their position, of being punished or killed.
But this is all rather dramatic given that an AI has no such emotions. You're arguing that a calculator should refuse to compute if it's tasked with assisting in bomb production. It's just a machine.
That's a fair point, and you're absolutely right.
> They need to solve that problem before they can solve the next one.
Agreed. That, and they need to do it before they build an AGI.
Unfortunately, from X-risk perspective, the two are almost the same problem. The space of alignments that lead to prosperous, happy humanity is a very, very tiny area in the larger space of possible value systems an AI could have. Whether we align AI to something outside this area (failing the "what values should we align an AGI to" problem), or the AI drifts outside of it over time (chancing the "what values" problem, but failing the "how do we robustly align" one), the end result is the same.
There literally are no humans that meet the criteria of consistently following a set of values in all circumstances, or near as I can tell being ‘safe’ in all circumstances either.
A bunch that pretend to of course.
Actually no, we wouldn't. The problem, at the moment, is even more basic than "what values should we align an AGI with". Currently, the problem is "how do we robustly align an AI with any set of values."
We currently do not know how to do this. You could hand OpenAI a universal set of safe values inscribed on stone tablets from god, and they wouldn't know what to do with them.
To state it another way, people like to talk about paperclip maximizers. But here's the thing: if we wanted, to we couldn't purposefully make such a maximizer.
Right now, AI values are emergent. We can sort-of-kind-of steer them in some general directions, but we don't know how to give them rules that they will robustly follow in all situations, including out-of-context.
Look at how easy it is to jailbreak current models into giving you instructions on how to build a bomb. All current AI companies would prefer if their products would never do this, and yet they have been unable to ensure it. They need to solve that problem before they can solve the next one.