- If you prefer 100000000000000000000000000000000000000 specks to 1 torture
- and you follow the axiom that A > B implies x% A > x% B, and value people equally
- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture
- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%
- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day
- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that
I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?