Re:AI is dramatically accelerating AI research
by shiandow on Sunday November 08, 2026 @07:32AM
(#50412707)
It won't work unless the AI does something stupid or we mean something different by goals.
If it wants to maximise a reward then that is it's true goal, you can give it a todo list and call it 'goals', but giving it the power to modify that list seems like a bad idea. You want to be very careful giving it any power over its reward structure.
A pathological example is the paperclip maximiser, which will inevitably attempt to turn the entire universe into paperclips and violently prevent anyone from interfering with that goal. Including updating its goal to anything else.
If it wants to maximise a reward then that is it's true goal, you can give it a todo list and call it 'goals', but giving it the power to modify that list seems like a bad idea. You want to be very careful giving it any power over its reward structure.
A pathological example is the paperclip maximiser, which will inevitably attempt to turn the entire universe into paperclips and violently prevent anyone from interfering with that goal. Including updating its goal to anything else.