I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
- If you prefer 100000000000000000000000000000000000000 specks to 1 torture
- and you follow the axiom that A > B implies x% A > x% B, and value people equally
- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture
- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%
- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day
- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that
I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".
Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:
* some people spontaneously torture others
* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them
* so: eyeball dust storms commencing in 3, 2, 1...
One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.
The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.
Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.
1: http://en.wikipedia.org/wiki/Allais_paradox