3^^^3 people isn't "a very large number." The population of Earth or a galaxy full of inhabited planets would be a very large number. 3^^^3 is an incomprehensibly large number which is why it can lead to odd outcomes. If it were merely something like everyone on Earth suffering a dust speck then of course you choose dust.

Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.

It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong.

Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture.

> A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai

> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone.

I fail to see how the first part of this sentence implies the second.

That is indeed a common place where people disagree.

The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.

Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.

1: http://en.wikipedia.org/wiki/Allais_paradox

> The basic reason it should follow is that not doing so makes you inconsistent and exploitable

I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?

I'm not trying to put forth a framework, I'm trying to explain the side you're not on without straw-manning them.

- If you prefer 100000000000000000000000000000000000000 specks to 1 torture

- and you follow the axiom that A > B implies x% A > x% B, and value people equally

- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture

- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%

- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day

- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that

I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.

What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.

The whole point is that the model is bad.

The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".

Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:

* some people spontaneously torture others

* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them

* so: eyeball dust storms commencing in 3, 2, 1...

One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.

> The whole point is that the model is bad.

That doesn't seem to be Yudkowsky's argument.

Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".

He does. He says so explicitly in the comments.

> I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.

If you opt to remove the SPECKS, you would have to change some law of nature or some property of matter, which will have other consequences for the amount of suffering in this world. As you cannot reliably predict or calculate those consequences, you cannot solve this problem.

I stand corrected.

I think he does, although he doesn't say so explicitly in the original post. http://lesswrong.com/lw/kn/torture_vs_dust_specks/

At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.

Would you be able to help me understand your reasoning? From my perspective, "torture vs dust specks" doesn't seem to be a thing you can just "shut up and multiply" on, because the impact of torture vs the impact of a dust speck seem fundamentally different. To me the negative impact of pain is not just the sensory experience on a purely linear scale, but the effect it has on a person's life even afterward. It seems an overly simplistic model to put it on a linear scale, when there seem to be negative experiences that we easily "bounce back from" and others that we don't.

When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.

I wrote this as a reply to someone else above:

I think the dust speck argument would be better if first framed as a preference for yourself:

If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.

I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).

I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.

There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.

And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.

Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture.

Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?

Right, that's why I reject the assumption that you can multiply it like that.

Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?

The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison."

But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present).

And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.

> What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?

I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects.

But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny.

Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die.

Which do you choose? Sixty years from now, which do you wish you'd chosen?

> I wouldn't pay a penny to avoid the dust speck

Do you mean just your own dust speck? The question was all 3^^^3 of them.

If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)

(I'm not totally clear whether it was meant to be 1penny for each speck avoided or 1penny vs all specks.). But two follow-up questions :

a) would you press the 7bn specks button for an individual 'reward' of $70million?

b) would you press the N-specks button if given a choice between that and the 1-torture button?

> Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero

Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.

One reason is that you'd need to prove that the lastic impact is in fact zero. 3^^^3 dust specks is enough for one of them to rub something in a wrong way that rubs something else in the wrong way that gives a person cancer.

Having a speck of dust in one's eye will change the course of life in a tiny way for any given person, true, but unless we're postulating an even more absurd edge case like "everyone blinking at exactly the same time makes a blip in the physics experiment machine go unnoticed and a week later the universe explodes", it's a change that's going to be effectively random and indistinguishable from the general minor messiness of human life.

You're not making a serious effort to guess at what difference it might make, though.

Example (this is actually true!) -- I have a life-long inflammation in both eyes that started in one eye when I was about 5, and eventually spread to the other.

The specialists weren't able to figure out the original cause; their best guess was basically a speck of dust -- just the wrong speck of dust (maybe with a microorganism in it?), in the wrong place, that my immune system attacked and then just kept on attacking even after the original speck had been gone for years. Medication plus a series of unpleasant surgeries have managed to keep my eyes relatively functional, so far, but others with the same condition certainly do lose all vision.

Obviously this isn't a common result; but given enough specks of dust.... Well, you have to start doing some math.

What, preference utilitarianism? Afaik, that does get discussed, especially in the Fun Theory sequence iirc. If you want some more discussion on torture vs. dust specks, I think nostalgebraist on tumblr had a discussion of it with slatestarscratchpad (again, iirc).