Re:AlphaGo beats Lee Sedol 3-0 [video]
by anonaoeu on Saturday March 12, 2016 @04:13AM
(#11272053)
If an AI has any motivation at all, say, to make paperclips as efficiently as possible, then any threat to its existence is a threat to its objective function - namely, to create paperclips. A hyper-intelligent entity who is instructed to optimize for paperclips created will therefore proactively remove threats to its existence (i.e. its paperclip-creating functionality) and might possibly turn the entire solar system into paperclips within a few years if its objective function isn't carefully determined.
[ Parent
| on HN ]
Such an entity would not be hyper-intelligent. It would be idiotic. One huge hole for me in the paperclip argument is that an AI capable of that kind of power would not be stupid enough to misinterpret a command - it would be intelligent enough to infer human desires.
Re:AlphaGo beats Lee Sedol 3-0 [video]
by sbierwagen on Saturday March 12, 2016 @12:09PM
(#11273543)
Yeah, but why would it want to? I can perfectly infer the values of an earthworm, but I don't dedicate all my resources to making worms happy.
Of course it would. But, it's not programmed to care about what you meant to say. It will gladly do what it was mis-programmed to do instead. You can already see this kind of trait in humans, where instinct is mis-aligned with intended result. Such as procreation for fun + birth control.
You're making the assumption that human desires would matter to an AI.
Sure it would. It just wouldn't be friendly to you.
Re:AlphaGo beats Lee Sedol 3-0 [video]
by oneeyedpigeon on Saturday March 12, 2016 @04:24AM
(#11272090)
Always check your loop invariants very carefully.