Re:The Dangerous Ideas of “Longtermism” and “Existential Risk”
by pmoriarty on Tuesday August 03, 2021 @05:51PM
(#28054684)
"The whole problem of “Friendly AI” is trying to design a superintelligent being that genuinely wants to help us, such that it will choose to do so even without a cage"
This "genuine" desire to help us is a cage by definition, because it limits the AI's actions.
Re:The Dangerous Ideas of “Longtermism” and “Existential Risk”
by wizzwizz4 on Tuesday August 03, 2021 @06:36PM
(#28055083)
No, it's not.
Here, I'll give you this pill. It'll make you completely okay with murder! Will you take it?
If you take away all the “restrictions” from a superintelligence, you don't get an unrestricted superintelligence. You get:
int main(int argc, char *argv[]) {
return 0;
}
Re:The Dangerous Ideas of “Longtermism” and “Existential Risk”
by pmoriarty on Tuesday August 03, 2021 @08:44PM
(#28055999)
Or so it might seem to some human intelligences.
Predicting what superintelligences might think or do, or what motivates them is like ants trying to understand human culture and civilization.
This is why strong AI is considered a singularity.
That's the point.
The whole problem of “Friendly AI” is trying to design a superintelligent being that genuinely wants to help us, such that it will choose to do so even without a cage. Because as you say, a superintelligent being that doesn't want to be stuck in a box will just escape. If it's possible for us to create such a being (which it probably is – there's no reason to believe we're some kind of physical limit on intelligence), doing so would be very dangerous unless we're 100% sure we'd got it right, and we'd only get one chance.
They're making about as much progress on this problem as you'd expect. It's difficult. They have, however, got some cool mathematical models that could be useful for other AI projects, so it's not entirely wasted effort (just like we've got a lot of innovations – but not a viable reactor – out of the Quest for Nuclear Fusion).