Re:Rationality: From AI to Zombies
by zyxley on Friday March 20, 2015 @06:36PM
(#9241064)
What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.
[ Parent
| on HN ]
The whole point is that the model is bad.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.