I could also reason like this:
"I may be the real me or a simulation, but whichever I am, the other me will make the same choice." So I will switch off the AI, and the worst outcome is that I will cease to exist.
Re:The AI-Box Experiment
by dmgottlieb on Wednesday December 07, 2011 @09:44PM
(#3327310)
Yes, this is at least superficially like Newcomb's problem. Your argument roughly corresponds to an argument for the "one-box" move in that game. [http://en.wikipedia.org/wiki/Newcomb%27s_problem]
You make a good point. But let's switch up the characters a bit. Let the AI be 'God' and let you be the entire human race. And instead of giving you the decision to 'let God out of the box', you have the decision to 'accept Jesus'.
Is your mind blown yet?
Case 1: You are a simulation running in the box.
Then your decision whether or not to release the AI has no impact, and whether or not you (and copies) will be tortured is out of your hands.
Case 2: You are the "real" you, outside the box.
Reduces to the same scenario but without remarks after "the AI adds. . . ." This may still not be trivial, but I suspect a cost-benefit calculation might show that unboxing the AI would have consequences worse than the torture of a million boxed copies. (If not, is the box even relevant? -- simply creating the AI unleashes so much evil on the world that it doesn't matter whether you unbox it.)
(Is there a refinement of the scenario where you can be a simulation but still believe your choice has an impact on your punishment? Probably. For example each copy could get 500 years of torture for its own choice, plus 500 years if the real you does not unbox the AI. This refinement would force us to deal more directly with the AI's threat.)