Re:There’s No Fire Alarm for Artificial General Intelligence...
by amcknight on Tuesday June 09, 2020 @03:48PM
(#23470136)
An AGI with a small goal that it can wirehead to max out is not harmless by default. It can always add 9s to its confidence that it isn't being tricked and then secure itself while doing so.
So an AGI without a value system is (mostly) harmless. If the AGI has a defined value system, it's a lot clearer what that value system is. No "whoops gonna destroy the world to make paperclips" will happen.
(A not-fully general AGI might not be smart enough to defeat its own sensors, yet still be smart enough to do lots of harm. The typical example is grey goo, which isn't smart at all, yet plenty harmful.)