Keep in mind that the 3^^^3 dust specks are spread across 3^^^3 people and their annoyance can't be simply added together.
Why not?
EDIT: Serious question. If I save two lives that is twice as good as saving one, right? Why is this situation different?
Most people measure goodness and badness by how they feel about it, and for anyone who feels a significant amount of badness (grief, anger, whatever) at the death of a single person, it is physiologically impossible for them to feel a million times worse about the death of a million people. It's intuitively obvious to most people, therefore, that suffering, annoyance, life-saving, and so on, are not additive: they just have to check how they feel about the situation to know that.
In order to suggest that they are additive, and that N "people annoyed" can outweigh M "people suffering", you have to first convince someone that how their own internal measurement of goodness (how they feel about it) is not as accurate as some external measurement.
Which is true. But requires a willingness to counter one's own intuition when encountering contradictory evidence. Unfortunately the type of person that does that is uncommon.
"The only way to rectify our reasonings is to make them as tangible as those of the Mathematicians, so that we can find our error at a glance, and when there are disputes among persons, we can simply say: Let us calculate, without further ado, to see who is right." -Leibniz
Unfortunately we retain some ignorance on the correct nature of utility functions (finite? time-preference adjusted? etc.), and terminal values for humans are demonstrably arbitrary.
... then LW ends up with Roko's Basilisk.
Really, you're using that as your answer to "I don't disagree with what you actually said, but your choice of analogy suggests that you believe that questions of morality are settled and have obvious, objective answers." You can prove anything if you first make it an axiom.
You can't seriously claim that utilitarianism accurately captures human moral intuitions. Variations on the Repugnant Conclusion occur immediately to anyone told about utilitarianism, and are discussed in first-year philosophy right there when utilitarianism is introduced.
LessWrong routinely has discussion articles showing some ridiculous or horrible consequence of utilitarianism. The usual failure mode is to go "look, this circumstance leads to a weird conclusion and that's very important!" and not "gosh, perhaps naive utilitarianism taken to an extreme misses something important."
I think these are the two points that those skeptical of utilitarianism have trouble with: it's exactly that it doesn't seem to match experience that started this thread. Additionally, it doesn't actually seem to work well in practice: http://econlog.econlib.org/archives/2014/07/the_argument_fr_...
Each individual has a tolerance of what they can comfortably cope with. If 3^^^3 people were all experiencing a pain that is below that tolerance, nobody would be prevented from happiness. However in the other situation, the tortured individual clearly would be.
Empathic "strangers" help others, in many contexts, often at personal risk.
That's what separates humans from singularitarian quantum computing devices.
First day, let a speck of dust enter your eye, at noon. Before sleeping, write down how you feel about that event.
Next day, rip your balls off at noon. Before sleeping, write down how you feel about that.
Now notice that 1/8e9 of 3^^^3 is more people than have ever lived.
I don't think this is well enough established, though it seems plausible.
If time does play a role, you can spread out the 3^^^3 dust specks. (Obviously, we don't realistically have enough time, but then we don't realistically have enough space either).
If you personally torture someone, would that have any effect on you? Is that good, bad, utilitarian?
Is there a recursive relationship between observers and observed systems? If so, does the intention of an observation matter?
Having either my friend have dust in his eye or some stranger tortured are outcomes with less utility in my book than the counterfactual default. Actions which lead to these outcomes are bad, actions which prevent or provide restitution are good.
I don't know what you are getting at with a recursive relationship between observers and observed systems.
I'm asking if emotion (e.g. desire to help friend) has higher or lower utility.
> recursive relationship berween observers and observed systems
See page 2 onward in this cybernetics paper
http://www.nomads.usp.br/pesquisas/design/objetos_interativo...
> His core beliefs include that the obvious goal of the human race is immortality
"We should be allowed to live as long as we want to." Emphasis on the want. I don't find that very controversial.
> that it is justifiable to kill a man if it would remove a spec of dust from the eye of every other person on Earth
No, the setup was one person's torture for 50 years (not death), or specks of dust in the eyes of 3^^^3 people. How much is 3^^^3 people?
"3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know." [1]That's a monumentally different setup than mere 7 billion humans on Earth. And it's making a point about the cold cruel calculus of moral utilitarianism, a very common and popular ethical position. Your disgust is precisely his rhetorical point...
(Edit: ...about human heuristics and biases getting in the way of the moral calculus of utilitarianism. If you are a utilitarianist, there is some cutoff of X independent people -- for some very large X -- for which it is better to save that many people a slight inconvenience at the expense of 50 years of torture for one individual. This follows straight from the math of finite utilitarianism. Yudkowsky's position of trusting the math over intuition may not be intuitive for most people, but I'd be surprised if a HN reader did not agree at least with the methodology.)
[1] http://lesswrong.com/lw/kn/torture_vs_dust_specks/