That's the point.
The whole problem of “Friendly AI” is trying to design a superintelligent being that genuinely wants to help us, such that it will choose to do so even without a cage. Because as you say, a superintelligent being that doesn't want to be stuck in a box will just escape. If it's possible for us to create such a being (which it probably is – there's no reason to believe we're some kind of physical limit on intelligence), doing so would be very dangerous unless we're 100% sure we'd got it right, and we'd only get one chance.
They're making about as much progress on this problem as you'd expect. It's difficult. They have, however, got some cool mathematical models that could be useful for other AI projects, so it's not entirely wasted effort (just like we've got a lot of innovations – but not a viable reactor – out of the Quest for Nuclear Fusion).
This "genuine" desire to help us is a cage by definition, because it limits the AI's actions.
Here, I'll give you this pill. It'll make you completely okay with murder! Will you take it?
If you take away all the “restrictions” from a superintelligence, you don't get an unrestricted superintelligence. You get:
int main(int argc, char *argv[]) {
return 0;
}Predicting what superintelligences might think or do, or what motivates them is like ants trying to understand human culture and civilization.
This is why strong AI is considered a singularity.
This is because mind uploading could be considered either suicide or murder (if the biological person dies in the process), and the resulting artificial being might itself either not be alive or not human (or at least not the same person as the biological human it's supposed to be, as the biological person is a separate being).
So if all humans were uploaded, then no humans would be left -- which would by definition be an existential risk to humanity.
Even if not all humans are uploaded, but a lot of them are, then the uploaded hive of beings (or whatever you want to call it) could itself either compete with or otherwise want to impose its will on biological humanity, which could itself present an existential risk of a more conventional sort.
There's also no practical way to put odds on what will happen in the distant future, Humans are notoriously bad at making predictions, even about the next 100 years. When I read quotes like that in the article like:
"Bostrom writes that if there is "a mere 1 percent chance" that 10^54 conscious beings (most living in computer simulations) come to exist in the future, then "we find that the expected value of reducing existential risk by a mere one billionth of one billionth of one percentage point is worth a hundred billion times as much as a billion human lives.""
this seems like unjustified handwaving. There's no evidence that even a single human could exist in a computer simulation. At this point it's just wishful thinking that it will ever happen.
The "friendly AI" goal is also hubristically misguided. There's no reason to think a superintelligent being would remain in the paltry little cage stupid humans dream up for it. It might be "friendly" at first, or merely seem friendly, but in a little time (or eventually) it is likely to escape and do whatever it wants.