News for hackers. Stuff that matters.
Re:There’s No Fire Alarm for Artificial General Intelligence... by someguyorother on Thursday June 04, 2020 @05:33AM (#23414493)
You're going to have to implement a value system into the AGI if it is a fully general AGI. Otherwise, it will just use its exceptional intelligence to defeat its own sensors, take control of its objective function, and essentially wirehead itself.

So an AGI without a value system is (mostly) harmless. If the AGI has a defined value system, it's a lot clearer what that value system is. No "whoops gonna destroy the world to make paperclips" will happen.

(A not-fully general AGI might not be smart enough to defeat its own sensors, yet still be smart enough to do lots of harm. The typical example is grey goo, which isn't smart at all, yet plenty harmful.)

Re:There’s No Fire Alarm for Artificial General Intelligence... by amcknight on Tuesday June 09, 2020 @03:48PM (#23470136)
An AGI with a small goal that it can wirehead to max out is not harmless by default. It can always add 9s to its confidence that it isn't being tricked and then secure itself while doing so.