The overlap between the Effective Altruism community and the Rationalist community is extremely high. They’re largely the same people. Effective Altruism gained a lot of early attention on LessWrong, and the pessimistic focus on AI existential risk largely stems from an EA desire to avoid “temporal-discounting” bias. The reasoning is something like: if you accept that future people count just as much as current people, and that the number of future people vastly outweighs everyone alive today (or who has ever lived), then even small probabilities of catastrophic events wiping out humanity yield enormous negative expected value. Therefore, nothing can produce greater positive expected value than preventing existential risks—so working to reduce these risks becomes the highest priority.

People in these communities are generally quite smart, and it’s seductive to reason in a purely logical, deductive way. There is real value in thinking rigorously and in making sure you’re not beholden to commonly held beliefs. But, like you said, reality is complex, and it’s really hard to pick initial premises that capture everything relevant. The insane conclusions they get to could be avoided by re-checking & revising premises, especially when the argument is going in a direction that clashes with history, real-world experience, or basic common sense.

Intelligence and rational thought is useful, but like any strategy it has its tradeoffs and limitations. No amount of intelligence can overcome the chaos of long time horizons, especially when we're talking about human civilization. IMHO it's reasonable to pick a long-term problem/risk and focus on solving it. But it's pure hubris to think rationality will give you anything approaching high confidence of what the biggest problems and risks actually are on a 20-50 year time horizon, let alone 200-500 years or longer.

The whole reason we even have time to think this way is because we are at the peak of an industrial civilization that has created a level of abundance that allows a lot of people a lot of time to think. But the whole situation that we live in is not stable at all, "progress" could continue, or we could hit a peak and regress. As much as we can see a lot of long-term trajectories (eg. peak oil, global warming), we really have no idea what will be the triggers and inflection points that change the social fabric in ways that are unforeseeable and quickly invalidate whatever prior assumptions all that deep thinking was resting upon. I mean 50 years ago we thought overpopulation was the biggest risk, and that thinking has completely flipped even without a major trajectory change for industrial civilization in that time.

I think one can levy a much more specific critique of rationalism: rationalism is in some sense self-defeating. If you are rational you will necessarily conclude that the fundamental dynamic that drives the (interesting parts of) the universe is Darwinian evolution, which is not rational. It blindly selects for reproductive fitness at the expense of all else. If you are a gene, you can probably produce more offspring in an already-industrialized environment by making brains that lean more towards misogyny and sexual promiscuity than gender equality and intellectual achievement.

The real conflict here is between Darwinism and enlightenment ideals. But I have yet to see any self-styled Rationalists take this seriously.

I always liken this to that we’re all asteroids floating in space. There’s no free will and everything is determined. We just see the whole thing unfold from one conscious perspective.

Emotionally I don’t subscribe to this view. Rationally I do.

My critique for rational people is that they don’t seem to fully take experience into account. It’s assumptions + rationality + experience/data + whatever strong inclinations one has that seems to be the full picture for me.

> no free will

That always seemed like a meaningless argument to me. To an outside observer free will is indistinguishable from a random process over some range of possibilities. You aren’t going to randomly go to sleep with your hand in a fire, there’s some hard coded biology preventing that choice but that only means human behavior isn’t completely random, hardly a groundbreaking discovery.

At the other end we have no issues making an arbitrary decision where there’s no way to predict what the better choice is. So what exactly does free will bring to the table that we’re missing without it? Some sort of mystical soul, well what if that’s also deterministic? Unpredictability is useful in game theory, but computers can get that from a hardware RNG based on quantum processes like radioactive decay, so it doesn’t mean much.

Finally, subjectively the answer isn’t clear so what difference does it make?

> That always seemed like a meaningless argument to me.

Same as that is not the lived experience. I notice that I care about free choice.

The idea that there's no free will may be a pessimistic outlook to some but to me it's a strictly neutral one. It used to be a bit negative, until I looked more closely that there's a difference between looking at a situation objectively and having a lived experience. When it comes to my inclinations and how I want to live life, lived experience takes precedence.

I don't have my thoughts sharp on it, but I don't think the concept even exists philosophically, but I think that's also what you're getting at. It's a conceptual remnant from the past.

"Free choice" is the first step towards the solution to this paradox: free will is what a deterministic choice feels like from the inside. The popular notion of free will is that our decisions are undetermined, which must imply that there is a random element to them.

But though that is the colloquial meaning, it doesn't line up with what people say they want: you want to make your choice according to your own reasons. You want free choice. But unless your own reasoning includes a literal throw of the dice, your justifications deterministically decide the outcome.

"Free will" is the ability to make your own choices, and for most people most of the time, those choices are deterministic given the options and knowledge available. Free will and determinism are not only compatible, but necessarily so. If your choices weren't deterministic, it wouldn't be free will.

This is the position that is literally called compatibilist.

But when you probe people, while a lot of people will argue in ways that a philosopher might call compatibilist, my experience is that people will also strongly resist the notion that the only options are randomness and determinism. A lot of people have what boils down to a religious belief in a third category that is not merely a combination of those two, but infuses some mysterious third options where they "choose" that they can't explain.

Most of the time, people who believe there is no free will (and can't be), like me, take positions similar to what you described, that - again - a proponent of free will might describe as compatibilist, but sometimes we oppose the term for the reason above: A lot of people genuinely believe in a "third option" for choices are made.

And so there are really two separate debates on free will: Does the "third option" exist or not, and does "compatibilist free will" exist or not. I don't think I've ever met anyone who seriously disagrees that "free will" the way compatibilists define it exists, so when compatibilists get into arguments over this, it's almost always a misunderstanding...

But I have met plenty of people who disagree with the notion that things are deterministic "from the outside".

It is stronger than compatibilism. Compatibilism argues that free will and determinism are orthogonal. The argument I summarized is that free will is and must necessarily imply determinism..

I think that is a distinction without difference in as much as it's an excuse not to deal with it. But compatibilist "free will" must imply determinism unless some "magic" third alternative exists, because there isn't another option, and there is no evidence to suggest such a third alternative exists, so in practice every compatibilist I've had this discussion with have fallen back on arguing free will is compatible with determinism.

Definition doesn't tell all implications. In practice compatibilists do deeper analysis that reveals that determinism is required for free will.

Opposing the term gives a wrong result too, as people jump to hard determinism.

However "hard" your determinism, there is no support for the notion of agency, and that is all that matters. Without agency, free will is nothing but an illusion, with same moral consequences.

And agency as inner motivation exists and is determined by your character, otherwise it would be not yours.

This isn’t true. The position I argued for above is that agency derives from determinism because it is necessarily causal: you have agency because you make decisions in line with your goals. If you didn’t make decisions that were deterministically selected from your goals, you’d actually lack agency!

I'm a regular practitioner of magic, have written essays about it on Quora, and I can identify this mysterious third option as "the universe responding to your needs." You can use any number of religious terms to refer it to, like serendipity and the like, but none of them can capture the full texture of precisely how free will operates.

Approaching this subject from a rational perspective divorces you from subject and makes it impossible to perceive. You have to immerse yourself in it and one way to do that is magical practice. Having direct experience of the universe responding to your actions and mindset eventually makes it absurdly clear that the universe bears intelligence and it's in this intelligence that free will operates.

I'd never thought before now to connect magic this directly to free will. Thanks for the opportunity to think this through! If you're interested in a deeper discussion, happy to jump on a call.

lmao, magic? seriously?

You betcha.

I think a reasonable interpretation of the colloquial sense of incompatibilist free will is that people want to be (or have the experience that they are) their own causal origins or prime movers. That they originate an action that is not (purely) the effect of all other actions that have occurred, but in such a way that they decided what that action was.

From the outside, this is indistinguishable from randomness. But from the inside, the difference is that the individual had a say in what the action would be.

Where this tends to get tangled up with notions of a soul, I think, is that one could argue that such a free choice needs some kind of internal state. If not, then the grounds by which the person makes the choice is a combination of something that is fixed and their environment, which then intuitively seems to reduce the free-will process to a combination of determined and random. So the natural thing to do is then to assign the required "being-ness" (or internal state if you will) to a soul.

But there may exist subtle philosophical arguments that sidestep this dilemma. I am not a philosopher: this is just my impression of what commonsense notions of free will mean.

My point is that from the outside this doesn’t look like randomness at all, unless you are mistaking ignorance of their motives as a random oracle. If you can infer what set of goals drives their decision making, and the decision making process itself (e.g. ADHD brain vs careful considered action) you can very much predict their decisions. Marketing and PR people do this every single day. People don’t behave like random oracles, they behave like deterministic decision makers with complex, partly unknown goals so our predictions of their behavior are not always correct. That’s not the same thing as random.

Divorced from a religious context, it doesn't make any difference.

Which religious context, and why?

I have much less patience for C++ than I would in a world with free will.

Since there's no free will, outcomes are determined by luck, and what matters is how lucky we can make people through pit-of-success environments. Rust makes people luckier than C++ does.

I also have much less patience for blame than I do in a world with free will. I believe, for example, that blameless postmortems lead to much better outcomes than trying to pretend people had free will to make mistakes, and therefore blaming them for those mistakes.

You can get to these positions through means other than rejection of free will, but the most robust grounds for them are fundamentally deterministic.

If there is no free will, then all arguments about what should be done are irrelevant, since every outcome is either predetermined or random, so you have no influence on whether the project at work will choose Rust or C++. This choice was either made 13 billion years ago at the Big Bang, or it is an entirely random process.

Lack of free will doesn’t prevent logical arguments from seeming to work.

Depends on whether you consider facts or theory. Facts don't prevent logical arguments from seeming to work, but lack of free will is theory. When theory doesn't match facts, theory is wrong.

We have built systems that don’t have free will and respond to logical arguments, so no theory is required here.

Random processes can’t use logic.

Fuzzy logic deals with truth values between 0 and 1. You can for example map water temperatures in such systems without having arbitrarily important cutoff points.

Such system often deal with uncertainty quite well including random noise on their inputs. The output ends up a function of both logic and randomness, but can still be useful.

Agreed, I don’t believe a system like that can access the platonic realm of mathematical truths. It’s clear that an electron doesn’t carry the laws of physics with it as it travels.

Why not? The human brain is hardly a perfect system of logic but can emulate such.

Something is orchestrating the decisions that the brain executes in the conscious realm though. (As opposed to or in contrast to the mathematical realm and the realm of the physical laws). We are clearly surrounded by invisible realms, unless you believe electrons are carrying lookup tables of how to respond to invisible electric fields.

This is a strawman argument extended by those who rely on supernatural explanations. In reality, people's utterances and actions are part of the environment that determines future actions, just like everything else.

Sure, but that still doesn't matter: the fact that I wrote my previous comment is what caused you to write your response, but it's not like I had a choice to write that comment or some other: the fact that I wrote that comment, as well as everything that led to me writing it (conversations with teachers, my parents letting me watch English cartoons so I learned English, etc), were predetermined the moment the Big Bang happened, or they're just a quantum fluctuation.

What I'm saying is that there's no logical point to the concept "should" unless you have some concept of free will: everything that happens must happen, or is entirely random.

> If there is no free will, then all arguments about what should be done are irrelevant, since every outcome is either predetermined or random, so you have no influence on whether the project at work will choose Rust or C++.

This is not correct. Whether or not you have free will, stuff influences and is influenced by other stuff, so these arguments are not meaningless or worthless.

> This choice was either made 13 billion years ago at the Big Bang, or it is an entirely random process.

I had thought of this before, and what I had decided is that both of these are also independent of having free will. For example, if the initial state includes unknown and uncomputable transcendental numbers which can somehow "encode" free will and then the working of physics is deterministic, then it is still possible (although not necessarily mandatory) to have free will, even though it is deterministic.

People get emotional about free will because if you come to believe there is no free will it makes you question a lot of things that are emotionally difficult.

E.g. punishment for the sake of retribution is near impossible to morally justify if you don't believe in free will because it means you're punishing someone for something the had no agency over.

Similarly, wealth disparities can't be excused by someone choosing to work harder, because they had no agency in the "decision".

You can still justify some degree of punishment and reward, but a lack of free will changes which justifications are reasonable very substantially.

E.g. punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals, and that has emotionally difficult consequences. For example, for non-premeditated murders carried out out of passion rather than e.g. gang crimes, the odds of someone carrying out another is extremely low and the odds that the fear of a long prison sentence is an actual deterrence is generally low, and and so long prison terms are hard justifiable once vengeance is off the table.

And so holding on to a belief in free will is easier to a lot of people than the alternative.

My experience is that there are few issues where people so easily get angry than if you suggest we don't have free will once they start thinking through the consequences (and some imagined ones...).

I don't find the consequences very hard to bear:

For example

> E.g. punishment for the sake of retribution is near impossible to morally justify if you don't believe in free will because it means you're punishing someone for something the had no agency over.

and

> E.g. punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals

are simply logical to me (even without assuming any lack of free will).

So what is emotionally difficult about this, as you claim?

Then you're highly unusual (in a good way). Look at the amount of comments on social media with outcries over "too short" sentences for example, and the lack of political support for shortening sentences or improving prison standards.

I'm saying it's emotionally difficult to people because I've had this discussion many times over then last 30+ years and I've seen first hand how most people I have this conversation with tend to get angry and agitated over the prospect of not having moral cover for vengeance.

> Then you're highly unusual (in a good way). Look at the amount of comments on social media with outcries over "too short" sentences for example, and the lack of political support for shortening sentences or improving prison standards.

I live in Germany.

When I observe the whole societal and political situation in the USA from the outside, it seems to me that it is rather "two blocks where in each of these there is somewhat an internal consensus regarding a quite some political positions. On the other hand, each of these two blocks is actively fighting the other one."

On the other hand, for Germany, I would claim that the opinions in society rather consist of "lots of very diverse stances (though in contrary to the USA less pronounced on the extreme ends) on a lot of topics that make it hard to reach a larger set of followers or some consensus in a larger group, i.e. in-fighting about all kinds of topics without these positions forming political camps (and the fractions for different opinions can easily change when the topic changes)."

Thus, in the given example, this means for a person out-crying "too short" sentences on social media, you will very likely find one who is out-crying the opposite position.

I agree; they seem logical to me too, whether or not you have free will.

However, it would seem that not everyone believes that, though.

(It is not quite as simple as it might seem, because the situation is not necessarily always that clear, but other than that, I would agree that it is logical and reasonable, that punishment is only justified from the point of view of reducing offending and reoffending rates and only if it actually achieves those goals.)

If there is no free will, thoughts about free will are predetermined and so is punishment. The punishers don’t have agency either. You seem to say that punishers do have free will, but criminals don’t?

I didn't say anything about whether free will exists or not, actually. The comment was specifically worded to explain why some people react to coming to believe there is no free will.

But, sure, I personally do not believe in free will. I'm saying there is no rational basis for thinking anyone has free will ever. I'm saying there is no evidence to suggest free will is possible. In fact, I'll go so far as to say that believing in free will is a religious belief with no support.

But that doesn't mean that events does not have effects on what happens next, just that we don't have agency. That an IF ... THEN ... ELSE ... statement is purely deterministic for deterministic inputs does not mean that changing the inputs won't affect the outputs.

If you "choose" to lay down and do nothing because you decide nothing matters because you don't have free will, you will still lose your job and starve. That it wasn't a "true" "free" choice does not change the fact that it has consequences.

One of the consequences of coming to accept that free will is an illusion is that you need to come to terms with what that means for your beliefs about a wide range of things.

Including that vengeance which might seem moral to some extent if the person who did something to you or others had agency suddenly become immoral. But we still have the feelings and impulses. Reconciling that is hard for a lot of people, and so a lot of people in my experience when faced with a claim like the one I made above that we have no free will tend to react emotionally to the idea of the consequences of it.

Are there deterministic solutions to the three body problem? Or the double pendulum? Or can you tell the t° at any point on earth for say, a millisecond in, say, 6h (feel free to chose a prefered point and time)? And what precision could you realistically produce in that?

If there are non deterministic processes that can be proven to exist, and those interact with deterministic processes, doesn't it follow that the deterministic process becomes non deterministic (since the result of the interaction is necessarily non deterministic), and that it is not continually deterministic.

So - can you see how nothing can be deterministic other than in isolation (or thought experiment really)?

Edit0: typo

There are deterministic solutions to the three body problem or the double pendulum in Newtonian mechanics.

We can’t measure things to arbitrary precision due to quantum mechanics, but Philosophy isn’t bound by the actual physical universe we inhabit. Arbitrary physical models allow for the possibility of infinite precision in measurement and calculation resulting in perfect prediction of future states forever. Alternatively, you could have a universe of finite precision (think Minecraft) which also allows for perfect calculation of all future states from initial starting concepts.

I agree, and indeed there are solutions to chaotic systems - the problem being precision as you mentionned. To me the precision problem it is important : it reframes the "mechanical universe" as being way out of our grasp not because of our understanding but because of it's structure. You got me!

Not certain that philosophy is not bound by our universe - is that something you could elaborate (or lend a link) on?

To apply these hypotheticals to our universe implies (from my understanding) that the density of information present at any and all times since it's inception was present (while compressed) at it's creation/whatever - which I imagine I can find some proof of theoretical maximum information density and information compression compare that to the first state of the universe we can measure to have a better idea if it tracks.

> Not certain that philosophy is not bound by our universe - is that something you could elaborate (or lend a link) on?

I simply mean it’s happy to assume perfect information, perfect clones, etc. The trolly problem generally ignores the possibility that choosing a different track could with some probability result in derailment because the inherent question is simplified by design. We don’t need for the possibility of perfect cloning to exist to consider the ramifications of such etc.

I think I see a distinction in that a hypothetical universe with perfect information is pertinent precisely because it is comparable to our measurable universe and could be tested against.

I guess that's the point of any hypothetical, exploring a simplified model of something complex, but it's not easy to simplify the fabric of reality itself.

That’s more effective as an argument to get rid of the most extreme forms of punishment (eg drawn and quartered) not all forms of retribution.

In a world without free will crimes of passion are simply the result of the situation which means that person would always chose murder in that situation. People who would respond with murder in an unacceptably wide range of situations is an edge case worth consideration without free will. Alternatively if we want nobody to respond with murder in a crime of passion situation evolutionary pressure could eventually work even without free will.

> E.g. punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals, and that has emotionally difficult consequences. For example, for non-premeditated murders carried out out of passion rather than e.g. gang crimes, the odds of someone carrying out another is extremely low and the odds that the fear of a long prison sentence is an actual deterrence is generally low, and and so long prison terms are hard to justify once vengeance is off the table.

That’s assuming absolute certainty about what happened. Punishments may make sense as a logical argument even if it’s only useful in a subset of cases if you can’t be absolutely sure which case something happened to be.

Uncertainty does a lot to align emotional heuristics and logical actions.

Whether or not you have free will is not relevant, as I had described in other comments.

> In a world without free will crimes of passion are simply the result of the situation which means that person would always chose murder in that situation. People who would respond with murder in an unacceptably wide range of situations is an edge case worth consideration without free will.

This is a significant argument. However, there is also worth considering if that is actually accurate, and if such a situation will occur (in a case where whoever would be killed would not effectively protect themself from this).

> That’s assuming absolute certainty about what happened. Punishments may make sense as a logical argument even if it’s only useful in a subset of cases if you can’t be absolutely sure which case something happened to be.

It is true that you do not have absolute certainty, but neither should you arrest someone who is not guilty.

> Uncertainty does a lot to align emotional heuristics and logical actions.

In some cases, yes, but it is not always valid. But, even if it is, this does not mean that you should not consider it logically if you are able to do so.

“ E.g. punishment for the sake of retribution is near impossible to morally justify if you don't believe in free will because it means you're punishing someone for something the had no agency over.”

False, the punisher also has no will, so it doesn’t matter.

I think that whether or not you have free will is not so important when making these considerations.

Whether or not you have a choice and free will, you can influence and be influenced by other stuff, since that is how anything is doing.

> punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals

I do agree with that, and I think that whether or not you have free will is not significant. Being emotionally difficult is not what makes it good or bad in this case (and it also does not seem to be so emotionally difficult to me, anyways). Reducing reoffending rates is what is important.

(Another issue is knowing if they are actually guilty (you shouldn't arrest people who are not actually guilty of murder); this is not always certain, either.)

I also think that it should mean that prisoners should not be treated badly and that prison sentences should not be too long. (Also, they shouldn't take up too much space by the prisons, since they should have free space for natural lands and for other buildings and purposes, but that is not quite the same issue, though.)

However, there may be cases where a fine might be appropriate, in order to pay for damages (although if someone else is willing to forgive them then such a fine may not be required). This does not justify a prison sentence or stuff like that, though.

Also, some people will just not like them anymore if they are accused of murder, even if they are not put in prison and not fined. This is not the issue for police and legal things; it is just what it will be. And, if it becomes known, people who disagree with the risk assessment can try to avoid someone.

And, if someone does commit a crime again and may have opportunity to do again in future, then this can be considered as being wrong the first time and this time hopefully you can know better.

Randomness based free will still gives you a non-inevitable future.

So does a computer without free will acting on a physical RNG. Therefore it’s the RNG that matters not free will.

For naturalistic libertarians , free.will is.partly constituted by indtermimksm, not something entirely.

If you get down to the quantum level there is no such thing as objective reality. Our perception that the world is made of classical objects that actually exist at particular places at particular times and have continuity of identity is an illusion. But it's a really compelling illusion, and you won't go far wrong treating it as if it were the truth in 99% of real-world situations. Likewise, free will is an illusion, nothing more than a reflection of our ignorance of how our brains work. But it is a really compelling illusion, and you won't go far wrong treating it as if it were the truth, at least some of the time.

> If you get down to the quantum level there is no such thing as objective reality.

What do you mean by that? It still exists doesn't it? Albeit in a probabilistic sense that becomes non-probabilistic at larger scales.

I don't know much about quantum other than the high level conceptual stuff.

> It still exists doesn't it?

It's controversial, but here is the argument that the answer is "no": See https://flownet.com/ron/QM.pdf

Or if you prefer a video: https://www.youtube.com/watch?v=dEaecUuEqfc

That's non sequitur.

>Under QIT, a measurement is just the propagation of a mutually entangled state to a large number of particles.

eyeroll so it's MWI in disguise, but MWI is quantum realism. Illusion they talk about is that the observed macroscopic state is a part of the bigger superposition (incomplete observation). But that's dumb, even if it's a part of a bigger state, it's still real, because it's not made up, but observed.

> it's MWI in disguise

That's kind of like saying that GRW is Copenhagen in disguise. It's not wrong, but only because it's making the word "disguise" do some pretty heavy lifting.

> MWI is quantum realism

No, it isn't because it can't account for the Born rule. See:

https://blog.rongarret.info/2019/07/the-trouble-with-many-wo...

If you insist that MWI must mean "a discrete number of clearly separated worlds", then yes, such interpretation would have a problem with the Born rule.

(That is apparently the definition the author of the linked article uses, guessing by his reaction: "Wait, what??? There is no 'well defined notion of how many branches there are?'")

I can only say that I have never met a proponent of MWI who meant this.

I am the author.

> I can only say that I have never met a proponent of MWI who meant this.

What can I say? There are a lot of MWI proponents who profess to believe this. Here, for example, is Sean Carroll answering the question, "How many parallel universes are there?"

https://www.youtube.com/watch?v=7tQiy5iCX4o

Of course, he doesn't actually give a concrete answer, but he very strongly implies that the question has an answer, i.e. that the question is a meaningful one to ask, and that implies that the MWI does in fact mean that there is a discrete number of clearly separated worlds.

In fact, I challenge you find a single example of a prominent MWI proponent saying something in public (which is to say, in a public forum or a publication whose target audience is the general public) that even implies that the many-worlds of the MWI are not distinct, countable entities. I only know of one example, and it is very well hidden.

There is a more fundamental problem: if the MWI does not mean "a discrete number of clearly separated worlds" then it fails as an interpretation of QM, i.e. as a solution to the measurement problem. The whole point is that measurements appear to produce discrete outcomes despite the fact that the math says that everything is one big quantum superposition. If all you have to say about this is, "Yeah, it's all one big quantum superposition" then you have failed to solve the problem. You have simply swept the hard part under the rug.

> Of course, he doesn't actually give a concrete answer, but he very strongly implies that the question has an answer, i.e. that the question is a meaningful one to ask, and that implies that the MWI does in fact mean that there is a discrete number of clearly separated worlds.

In the video, Sean Carroll talks to a non-expert audience, so he must simplify some things, and then it is your or my guess about what the unsimplified version was supposed to be. He says something like: "we don't know, even whether it is finite or infinite, but if it is finite it is a very large number such as 10^10^123". But notice that he also uses as an analogy an interval from 0 to 1, which can be split to half as many times as you need.

You see this as him believing in discrete separated universes, of which there is a definite number (potentially infinite). Yes, that makes sense.

I see another possible understanding, that he is talking about "meaningfully different" universes, because that is what we care about on the macro level. To explain what I mean, imagine that we observe two particles. Any of them can be in a huge number of possible positions, moving in a huge number of possible directions, at a huge number of possible speed. But if we ask whether those two particles hit each other and transformed into another particle, that kinda collapses this huge possibility space into a "yes / no" question. Out of practically infinity, two meaningfully different options.

On a macro level, either the cat is alive or it is dead. Those are two meaningfully different states. If we focus on one particle in the cat's body, there is a continuum of where precisely that particle could be, and what momentum it has. So from the particle's perspective, there is a continuum of options. But from the cat's perspective, and the cat's owner's perspective, this continuum does not matter; unless it changes the macro state, i.e. the particle kills the cat, or at least maybe hits its neuron and makes it do something differently. So it seems possible to me that Sean Carroll talks about the number of worlds that are different from human perspective.

Then there is another problem in physics that we don't know how/whether the very space and time are quantized. We use the mathematical abstraction of a "real number" that has an infinite number of digits after the decimal dot, but of course that infinite number of digits can never be observed experimentally. We don't know. Maybe it is something like what Wolfram says, that on a deep level, spacetime is a discrete graph evolving according to some rules. If something like that would be the case, that would reduce the possible number of states in the universe, even on the micro level, to a huge but finite number. And the mixed state of the multiverse would consist of this finite number of branches, each of them assigned a tiny complex amplitude. So that's another way how things could get finite.

And I am saying this just as a random guy who never studied these things, I just sometimes read something on the topic, and some ideas feel to me like obvious consequences of the stuff that is "in the water supply". So I believe that if I see a solution to a problem, then if it makes sense, someone like Sean Carroll is 10000x more likely to notice the problem and the solution, and develop it much further than I ever could. Or when you make a survey, and a half or a third of people who study quantum physics for living say that some version of MWI seems like the correct interpretation to them, I don't believe there is a simple devastating argument against it that all of these people have simply missed.

> I am saying this just as a random guy who never studied these things

OK, well, let me tell you as a non-random guy who has studied these things extensively that the MWI is very commonly misrepresented. It is not a case of simplification for a lay audience, it is flat-out lying, at least most of the time. The math does not say that there are parallel universes. All the math tells you is that in order to recover the results of experiments you have to throw away some of the information contained in the wave function. MWI proponents interpret this by saying that the discarded information has to correspond to something real, and they call that thing "parallel universes". But there are three problems with this. First, the MWI does not explain the Born rule. Second, the math doesn't tell you whether or not the discarded parts of the wave function describe something real or not. It is possible that mathematical operation of discarding parts of the wave function actually corresponds to real physical phenomenon, i.e. that whatever is described by the discarded parts of the wave function actually ceases to exist. This is a tenable scientific hypothesis. It's not easy to actually make it work, but it can be done and has been done. It's called GRW collapse [1]. So anyone who tells you that the MWI is the only possible scientifically tenable interpretation of QM is lying. And anyone who leaves open even the possibility that the "parallel universes" contained in the wave function are discrete is also lying. The only MWI proponent I've ever seen being intellectually honest about this.David Deutsch in his book "The Beginning of Infinity" chapter 11.

The third problem with the MWI is something called the "preferred basis problem". This one is harder to describe succinctly, and some people claim it has been solved, but I don't agree with them. In a nutshell, all two-state QM experiments rely on some macroscopic apparatus to split a particle into a superposition of two states. But if you model the entire universe as a quantum system, this apparatus is itself a quantum system that can be in a superposition of states, so you can't say, "The polarizing beam splitter is aligned vertically or it is aligned horizontally" any more than you can say "the cat is alive or it is dead" without begging the question.

---

[1] https://en.wikipedia.org/wiki/Ghirardi%E2%80%93Rimini%E2%80%...

>Second, the math doesn't tell you whether or not the discarded parts of the wave function describe something real or not.

The math tells that there are no privileged parts of wave function.

>So anyone who tells you that the MWI is the only possible scientifically tenable interpretation of QM is lying.

Didn't you admit yourself that if MWI works it's a big deal and will kick the chair from under other interpretations?

> The math tells that there are no privileged parts of wave function.

That's true. But my senses tell me that there is a privileged part of the wave function, namely, the branch that I'm in.

The way I think about it nowadays is that QM is like a Necker cube. You can look at it in two different ways. You can take the God's-eye view and look at the entire wave function, or you can take the mortal's eye view and look at only a proper subset of the wave function (which is necessary in order to recover classical reality). But you can't do both at the same time. For my day-to-day life, I have no choice but to take the mortal's-eye view because I am a mortal. All of the things that matter to me depend on classical reality, and so depend on my suspension of disbelief and acting as if my branch of the multiverse is privileged, even if I can intellectually jump out of the system momentarily and recognize that the mortal's eye view is necessarily incomplete.

> Didn't you admit yourself that if MWI works it's a big deal and will kick the chair from under other interpretations?

That depends on what you mean by "works". If someone can derive the Born rule from the Schrodinger equation that will be a big deal, a slam-dunk Nobel prize. But no one has done it, and I'm pretty sure it can't be done. I'm pretty sure that the Born rule is an emergent property of our branch of the multiverse. I believe the same is true of the Second Law and even three-dimensional space. You can slice-and-dice the wave function to give you physical spaces with any number of dimensions, but three is the magic number that gives you atoms and stars and planets with stable orbits [1] and so on. So I'm pretty sure the Born rule can only be explained by the anthropic principle. There's probably a Nobel prize waiting for the person who turns that intuition into a theorem.

---

[1] https://physics.stackexchange.com/questions/50142/gravity-in...

>But my senses tell me that there is a privileged part of the wave function, namely, the branch that I'm in.

You could also sense that your location is privileged, because the observable universe is neatly centered at it, but science will prioritize Copernican principle over your senses.

>But you can't do both at the same time.

This doesn't match what you do. Tracing extracts mortal's-eye view from God's-eye view, so in God's-eye view you have both.

>depend on my suspension of disbelief and acting as if my branch of the multiverse is privileged

If your branch exists, it's sufficient for your day to day life, there's not much else to disbelieve. There's no need for it to be privileged. Do you worry that Earth isn't more privileged than Mars?

>But no one has done it, and I'm pretty sure it can't be done.

But quantum physics doesn't allow it. It's quantitative science where all observed phenomena are computable. If they aren't computable, then quantum physics doesn't predict them and thus diverges from observation. And Schrödinger equation is how predictions are made, collapse and measurement only act on what already exists before them and don't create anything new. So if Born rule is an observed phenomenon, it must be computable from Schrödinger equation. Also if Born rule holds with certainty, then it's a pure state, and observation won't do anything to it, so Born rule can't be created by observation.

>There's probably a Nobel prize waiting for the person who turns that intuition into a theorem.

This was argued by Max Tegmark https://arxiv.org/abs/gr-qc/9702052 I thought it's a famous diagram https://en.wikipedia.org/wiki/File:Spacetime_dimensionality....

> You could also sense that your location is privileged, because the observable universe is neatly centered at it

Yes, that's right. This is not an easy problem to solve. This is why it took thousands of years for mankind to realize that the earth is not at the center of the universe. The difference between that and the MWI is that there is actual evidence against geocentrism. There is no evidence against my-branch-centrism. Not only that, but the theory itself predicts that there cannot possibly be any such evidence. So the MWI is self-defeating. The only way there could be evidence for it is if it's wrong.

> in God's-eye view you have both

Nope. The mortal's-eye view is fundamentally incompatible with the god's-eye view. This is the reason that the measurement problem is a thing in the first place.

> if Born rule is an observed phenomenon, it must be computable from Schrödinger equation

Only if the SE is a complete description of reality, and it manifestly is not.

(If you want to argue that the Born rule is not "an observed phenomenon" then I don't know what to tell you. Maybe go hang out with the flat-earthers and lunar landing denialists. You may find kindred spirits there.)

> This was argued by Max Tegmark

Yes, the 3-D space part. That is old news. It's the Born Rule that (AFAIK) no one has yet derived.

I think there's evidence for MWI and it's of the same character as evidence against geocentrism, you just question it, because evidence isn't airtight and geocentric prejudice is compelling because you reject Copernican principle and don't believe in relativity. But evidence against geocentrism isn't airtight either and can be questioned, and geocentrism can be hypothesized and is internally consistent, so why not, especially if you reject Copernican principle.

One branch interpretations are based on geocentric prejudice that the observer's state isn't changed much by observation (because observer doesn't feel change), and when the observer's state doesn't change much, we get geocentrism. But mathematics of quantum physics shows otherwise: the observer's state suffers decoherence and splits into macroscopic superposition, which is a big change and thus debunks assumption of unchanged observer's state. When observer's state changes significantly, observation becomes subject to relativity effect just like in case of spinning Earth.

>The only way there could be evidence for it is if it's wrong.

And what it means when there's no such evidence?

>The mortal's-eye view is fundamentally incompatible with the god's-eye view.

But then tracing must be fundamentally unable to extract mortal's-eye view from god's-eye view. What you say doesn't match what you do.

>If you want to argue that the Born rule is not "an observed phenomenon" then I don't know what to tell you.

I argue that Born rule is an observed phenomenon, and all observed phenomena are purely quantitative physical processes computable from Schrödinger equation, Born rule is the same, otherwise quantum physics wouldn't predict observation of Born rule.

Formally you might need measurement, but the trick is to convert the given problem into a problem of certainty, then measurement is trivial, and prediction is completely calculated from Schrödinger equation. Coincidentally Born rule is such a certain fact, so it doesn't matter if you measure it or not, measurement doesn't do much to certain facts, it's sufficient if you only calculate this certain fact and leave it as is without measuring it.

> observer doesn't feel change

It's not just that the observer doesn't feel change, it is that no experiment can demonstrate this change, not even in principle.

> tracing must be fundamentally unable to extract mortal's-eye view from god's-eye view

Why? Because that is manifestly not the case.

> Born rule is the same

Again, you are manifestly wrong. If someone had figured out how to derive the BR from the SE it would be Big News [1].

---

[1] https://blog.rongarret.info/2024/04/the-scientific-method-pa...

You got the preferred basis problem backward. It's one branch interpretations that have this problem, since the extant branch is absolute reality and its basis is absolute reality basis, hence the problem of privileged basis. In MWI all branches are relative and none is absolute reality, so their basis can't be thought of as absolute reality basis, branches and basis exist, but they are equal to others. This produces other criticism: universal wave function thus becomes spaghetti in which classical behavior is difficult to see when you look at it from outside, because the basis of classical behavior isn't privileged.

No, MWI has this problem too because it still has to account for the branch I am in.

The only interpretation that does not have this problem is the NCI because in that interpretation I am part of the fundamental ontology, at least some of the time.

It's a strange conclusion. You seemingly consider one measurement and expect to see Born rule, and when it doesn't manifest, then MWI is wrong? But Born rule doesn't manifest at sample size one in any interpretation, it manifests only in a long string of measurements. If you consider a long string of measurements, you will see Born rule as <Ψ|Born rule> = 1 - O(exp(-N)), which is basically a definition of empirical tendency.

Well, now I see that QIT isn't quite there. You say classical behavior emerges by tracing, mathematically, not as a physical process? In MWI classical behavior emerges as a physical process, not by tracing. That "look at part of the system (in which case you see classical behavior)" is provided by linear independence of different branches, so each observer naturally observes their branch from inside, and it looks isolated from other branches.

> You seemingly consider one measurement and expect to see Born rule

Huh??? No, of course not. The Born rule is about probabilities. It cannot manifest in a single measurement.

> classical behavior emerges by tracing, mathematically, not as a physical process?

No. The mathematical description of classical outcomes emerges by tracing, which is to say, by discarding information. The physical interpretation of that is left open.

> In MWI classical behavior emerges as a physical process

That's right. MWI commits to a physical interpretation of the math. But there is no scientific or philosophical justification for this, and in fact, when you dig into the details all kinds of problems emerge that are swept under the rug by its proponents. Nonetheless, many MWI proponents insist that it is the One True Interpretation, including some who really ought to know better.

> each observer naturally observes their branch from inside, and it looks isolated from other branches.

Yes, I know. But this doesn't solve the problem. In order to get a mathematical description of me I have to trace the wave function in my preferred basis, which is to say, I have to throw out all of the other branches. And this is not just a computational hack. It's mathematically necessary. Discarding information is the only way to get classical, irreversible processes (like measurement) out of the unitary dynamics of the wave function. So a reasonable interpretation of the math is that I exist only if parallel universes don't. And I'm pretty sure I exist.

I'm not telling you this because I expect you to accept it, merely to show you that the MWI is not self-evidently the One True Interpretation.

In the blog post you say that tracing lets you consider a subset of an entangled system in isolation from the rest of the system. That was consistent with MWI where states are isolated from each other and not discarded. Mathematically discarding isn't necessary, isolation is sufficient.

Sure. So? MWI is not mathematically untenable, it's just incomplete (because it can't account for the Born rule) and IMHO philosophically untenable because it requires that no experiment can demonstrate the existence of a fully isolated universe, i.e. a universe whose macroscopic configuration is different from ours [1]. This feature of the MWI is what I call an IPU -- an Invisible Pink Unicorn -- something that the theory insists exists despite the fact that the theory also requires it to be unmeasurable even in principle. If you want to believe it exists, fine. Just don't call it science, and definitely don't insist that anyone who doesn't accept it is being irrational. It's attitudes like that that turn Rationalism (with a capital R) into a cult.

---

[1] https://blog.rongarret.info/2009/04/on-shadow-photons-and-re...

(Note that I wrote this 16 years ago, so not everything is 100% accurate, but I stand by the central point.)

In MWI the Born rule is a tendency of statistics of a long string of measurements. You tried to get this statistics from one measurement, which didn't work. If you want to see how MWI produces the Born rule, you should calculate statistics of a long string of measurements and see that this statistics is asymptotically close to Born statistics.

Other branches can be demonstrated indirectly by 1) quantitatively verifying unitary dynamics, 2) indirectly observing branching, 3) demonstrating that other theories are wrong. Branches are just superposition, if you want to eliminate branches, you should eliminate superposition with pilot wave or superdeterminism or something like that. This kind of unobservability isn't unique to MWI, in general theory of relativity we observe only a part of the universe, the rest being beyond event horizon and is unobservable. Do you believe only observable part of the universe exists and beyond it nothing exists?

> You tried to get this statistics from one measurement

Huh??? When?

> Other branches can be demonstrated

Sure, that's just QM 101. What you cannot demonstrate experimentally, not even in principle, is the existence of other branches with different macroscopic configurations than our own. Such branches are IPUs.

>Huh??? When?

In the blog post you linked above:

>No, it isn't because it can't account for the Born rule. See:

>https://blog.rongarret.info/2019/07/the-trouble-with-many-wo...

>What you cannot demonstrate experimentally, not even in principle

I provided 3 ways to demonstrate it experimentally, even in principle, not sure what problem you have.

> In the blog post you linked above

Again, huh??? Where in that blog post do I try to "get this [sic] statistics from one measurement"?

> I provided 3 ways to demonstrate it experimentally, even in principle, not sure what problem you have.

No, you didn't. You apparently don't understand what is meant by "branches with different macroscopic configurations than our own" and I don't have time to explain it to you. Sorry. Go read up on decoherence, and then come back and describe an experiment that can demonstrate the existence of a fully decohered branch. You can't, because if you could it would by definition not be fully docohered.

>Where in that blog post do I try to "get this [sic] statistics from one measurement"?

In the discussion how different people place bets on A and B outcomes of experiment. Well, you didn't state clearly why you believe that MWI doesn't account for Born rule. MWI accounts for Born rule as statistics of measurements, and the discussion of bets is the closest this in that blog post to consideration of statistics of measurements, but that discussion seemingly considers one measurement, that's why it doesn't see statistics.

>Go read up on decoherence, and then come back and describe an experiment that can demonstrate the existence of a fully decohered branch.

It looks like a logical problem to me. You suggest that decoherence both produces and doesn't produce fully decohered branches? Violation of the law of excluded middle? If the law of excluded middle doesn't work, I don't think experiments can demonstrate anything.

> you didn't state clearly why you believe that MWI doesn't account for Born rule

That's what the whole post was about. The MWI doesn't account for the Born rule unless you add additional, questionable assumptions like branch indifference to the SE.

> You suggest that decoherence both produces and doesn't produce fully decohered branches?

No, that is not even remotely what I am saying. You are beginning to sound like a troll.

That blog post associates for me with the indirect observation of branching that I mentioned. Double slit experiment with one detector has a seemingly impossible phenomenon when the interference pattern disappears without measurement in our branch, so without branching this phenomenon can't be explained as "pattern disappears due to measurement", because measurement didn't happen. In MWI in this case the pattern disappears due to measurement in the other branch, that branch splits from our with the measured part of the photon and the pattern disappears in our branch. Our branch isn't privileged in this case, actually it's affected by the other branch, and the situation is symmetric: when measurement happens in our branch, pattern disappears in the other branch without measurement there. Well, technically if the other branch is destroyed by measurement there, the result for our branch is the same I guess. And if this phenomenon can't be explained without branching, then it's almost direct evidence for branching.

> Double slit experiment with one detector has a seemingly impossible phenomenon when the interference pattern disappears without measurement in our branch

Then you don't understand quantum mechanics at all. You should read this:

https://flownet.com/ron/QM.pdf

The TL;DR is that measurement and entanglement are the same phenomenon. A particle can become entangled with a detector even if the detector doesn't register anything.

But that is neither here nor there. Why do you get interference with no detectors? Your theory is that a detector at one slit is somehow paired with a "virtual detector" in a parallel branch at the other slit. But why would that "virtual detector" go away when the real detector is removed? Why is it never the case that there is a "virtual detector" at either slit unless there is a real detector at one of them?

Before measurement it's one detector, then it interacts with half photon, which is superposition |photon>+|no photon>, and the detector's state splits, the first state interacts with the |photon> state and measures it, nothing happens to the second state, because there's nothing to measure in the |no photon> state, but the first state splits away from the second state and leaves it alone, that's how the second state ends up alone with pure |no photon> state, but this really was done to it by the first state, the second state can't do it by itself. For this to happen in the second branch you need measurement to happen in the first branch, so when you are in the second branch, the first branch still needs to exist even though you don't see it, otherwise your branch won't be able to be as it is.

When you remove detector and start next measurement, you start with your one branch, branches from previous measurements don't affect it, the phenomenon happens during decoherence, nothing happens after it.

Your article explains this with branching without saying the word.

There is no such thing as a half photon.

It's a part of photon's state at the slit with detector, the other part is at another slit. It's superposition of photon near detector and no photon near detector.

A superposition is not "half a photon". You are talking nonsense.

It’s probabilistic at all length scales. For example our solar system may suddenly come undone according to simulations.

There is no local realism. That doesn't at all add up to all-in-the-head idealism.

That's true. There is a metaphysical reality "out there", but it is radically different from what we perceive. Hence: an illusion. Note that an illusion is emphatically NOT the same thing as a delusion. Illusions are real sensory experiences common to nearly all humans. They just happen not to correspond to reality.

How would you know? If all that's known is either known through the senses or drawn out by reason from what is known through the senses, then by declaring that sense data do not reflect reality, you've cut yourself off form the possibility of knowing reality altogether.

> How would you know?

Because that is the best explanation for what I observe.

> by declaring that sense data do not reflect reality, you've cut yourself off form the possibility of knowing reality altogether

That is true, but only in the uninteresting sense that I can never completely eliminate the possibility that I am living in the Matrix. So yes, it's possible that I'm wrong about the existence of objective reality. But if objective reality is itself an illusion, it's a sufficiently compelling illusion that I'm not going to go far wrong by acting as if it were real.

> That is true, but only in the uninteresting sense that I can never completely eliminate the possibility that I am living in the Matrix. [...] But if objective reality is itself an illusion, it's a sufficiently compelling illusion that I'm not going to go far wrong by acting as if it were real.

That seems squishy, as what constitutes "going far wrong" is not meaningful under skeptical assumptions.

A better stance is one of cognitive optimism that avoids the irrationality of skepticism. Skepticism is irrational, because it leads to incoherence, and because there is no rational warrant to categorically doubt the senses. For doubt to be rational, there must be a reason for it. To doubt without reason is not to be rational, but to be willful, and willful beliefs cannot be reasoned with; they are not the product of evidence or inference — and they certainly aren't self-evident — but rather the product of arbitrary choice. The logical possibility of living in the Matrix is no reason for doubting the senses, just as the logical possibility of there being poison in your sandwich is no reason for doubting you'll survive eating it.

The difference between our positions is that I begin from a position of natural trust toward the senses and toward reason as the only rational possibility and default. I have no choice but to reason well or to reason poorly. I recognize that my senses and my inferences can err, but it does not follow that they always err. Indeed, the very claim that they can err presumes I can tell when they do.

So, if my inferences lead me to a position that undermines their own coherence, then I must conclude that my inferences are wrong (including those that led me to adopt a certain interpretation of, say, scientific measurements).

> Because that is the best explanation for what I observe.

But if your explanation involves contradiction of what you observe, then that is not only not the best explanation, but no explanation at all! An explanation cannot deny the thing it seeks to explain. Thus, by denying the objective reality of what you perceive, you are barred from inferring that denial.

> what constitutes "going far wrong" is not meaningful under skeptical assumptions.

I can be more precise about this. It means that the predictions I make on the basis of this assumption are very likely to be correct.

> Skepticism is irrational

No, it isn't. The vast majority of my beliefs about the world are not a result of direct observations, but nth-hand accounts. I believe, for example, that the orbit of Mercury precesses, but not because I've ever measured it myself, but rather because I heard it from a source that I consider credible. But assessing the credibility of a source is hard and error-prone, especially nowadays. There is always the possibility that a source is mistaken or actively trying to deceive you. And even for things you observe first-hand there are all kinds of cognitive biases you have to take into account. So skepticism is warranted.

> I begin from a position of natural trust toward the senses

That will lead you astray because your senses are unreliable.

> if your explanation involves contradiction of what you observe

But it doesn't. At worst it involves a contradiction of what I think I observe.

It can't be that different, either, or our senses would be of no practical use.

That's not true. What our senses perceive (classical reality) is an emergent phenomenon of the underlying metaphysical truth (quantum mechanics). Those two things are about as radically different as you can get. That's why the measurement problem is a thing. But that doesn't mean our senses are of no practical use.

Look into Chaos Theory - the universe is not deterministic, you're good.

Or for a tldr look for the three body problem or try to find a solution to a double pendulum!

I'd like to believe that there is no such thing as free will, but I just can't decide.

“ There’s no free will and everything is determined.”

Objects without free will aren’t able to come to conclusions like this.

To the contrary, here's a series of essays on the subject of evolutionary game theory, the incentives created by competition, and its consequences for human wellbeing:

https://www.lesswrong.com/s/kNANcHLNtJt5qeuSS

"Moloch hasn't won" is a lengthy critique of the argument you are making here.

That doesn't seem to be on point to me. I'm not talking about being "caught in bad equilibria". My assertion is that rationalism itself is not stable, that the (apparent) triumph of rationalism since the Enlightenment was a transient, not an equilibrium. And one of the reasons it was a transient is that self-styled rationalists believed (and apparently still believe) that rationalism will inevitably triumph because it is rational, because it is in more intimate contact with reality than religion and superstition. But this is wrong because it overlooks the fact that what triumphs in the long run is simply reproductive fitness. Being in contact with reality can be actively harmful to reproductive fitness if it leads you to, say, decide not to have kids because you are pessimistic about the future.

> Being in contact with reality can be actively harmful to reproductive fitness if it leads you to, say, decide not to have kids because you are pessimistic about the future.

The fact that you can write this sentence, consider it to be true, and yet still hold in your head the idea that the future might be bad but it's still important to have children suggests that "contact with reality" is not a curse.

> it overlooks the fact that what triumphs in the long run is simply reproductive fitness.

Why can't that observation be taken into account? Isn't the entire point of the approach accounting for all inputs to the extent possible?

I think you are making invalid assumptions about the motivations or goals or internal state or etc of the actors which you are then conflating with the approach itself. That there are certain conditions under which the approach is not an optimal strategy does not imply that it is never competitive under any.

The observation is then that rationalism requires certain prerequisites before it can reliably out compete other approaches. That seems reasonable enough when you consider that a fruit fly is unlikely to be able to successfully employ higher level reasoning as a survival strategy.

> Why can't that observation be taken into account?

Of course it can be. I'm saying that AFAICT it generally isn't.

> rationalism requires certain prerequisites before it can reliably out compete other approaches

Yes. And one of those, IMHO, is explicit recognition that rationalism does not triumph simply because it is rational, and coming up with strategies to compensate. But the rationalist community seems too hung up on things like malicious AI and Roko's basilisk to put much effort into that.

This argument proves too much. If rationalism can't "triumph" (presumably over other modes of thought) because evolution makes moral realism unobservable, then no epistemic framework will help you - does empirically observing the brutality of evolution lead to better results? Or perhaps we should hypothesise that it's brutal and then test that prediction against what we observe?

I'm sympathetic to the idea that we know nothing because of the reproductive impulse to avoid doing or thinking about things that led our ancestors to avoid procreation, but such a conclusion can't be total because otherwise it is self defeating because is is contingent on rationalist assumptions about the mind's capacity to model knowledge.

The point being made is that rationalism is a framework. Having a framework does not imply competent execution. At lower levels of competence other strategies win out. At higher levels of competence we expect rationalism to win out.

Even then that might not always be the case. Sometimes there are severe time or bandwidth or energy or other constraints that preclude carefully collecting data and thinking things through. In those cases a heuristic that is very obviously not derived from any sort of critical thought process might well be the winning strategy.

There will also be cases where the answer provided by the rational approach will be to conform to some other framework. For example where cult type ingroup dynamics are involved across a large portion of the population.

> Having a framework does not imply competent execution,

Exactly right. It is not rationalism per se that is the problem, it is the way that The Rationalists are implementing it, the things they are choosing to focus their attention on. They are worried about things like hostile AI and Roko's Basilisk when what they should be worried about is MAGA, because that is not being driven by rationalism, it is being driven by Christian nationalism. MAGA is busily (and openly!) undermining every last hint of rationalism in the U.S. government, but the Rationalist community seems oddly unconcerned with this. Many self-styled Rationalists are even Trump supporters.

I hesitate to nitpick, but Darwinism (as far as I know) is not really the term to use because Darwin's theory was limited to life on earth. Only later was the concept generalised into "natural selection" or "survival of the fittest".

I'm not sure I entirely understand what you're arguing here, but I absolutely do agree that the most powerful force in the universe is natural selection.

The term "Darwinian evolution" applies to any process that comprises iterated replication with random mutation followed selection for some quality metric. Darwin himself would not have defined it that way, but he still deserves the credit for being the first to recognize and document the power of this simple process.

The modern understanding of Darwin's theory (even the original theory, not necessarily neo-Darwinian extensions of it) apply to the origins of life and non-biological systems as well. Darwin himself was largely concerned with biology and restricted his published writings to that topic, but even he saw the application to the origin of life, and implications for religion. Even if he hadn't, we generally still use the discoverer's name to a theory even when applied to a domain outside their original area of concern.

The term "survival of the fittest" predates Darwin's Origin of Species, and was adopted by Darwin within his lifetime, btw.

You got Darwinism exactly backwards. Darwinism and nature do not select like an algorithm. There is no cost function in reality and no population selection and reproduction algorithm. What you're seeing is the illusion of selection due to selection bias.

If gender equality and intellectual achievement don't produce children, then that isn't "darwinism selecting rationality out". You can't expect the continued existence of finite lifespan organisms if there are no replacement organisms. Raising children is hard work. The people who believe in gender equality and intellectual achievement made the decision to not want more of themselves, particularly when their belief in gender equality entails not wanting male offspring. The alternative is essentially freeloading and expecting others, who do not share the beliefs, to produce children for you and also to teach them the "enlightened" belief of forcing "enlightened" beliefs onto others (note the circularity, the initial conditions are usually irrelevant and often just a fig leaf to perpetuate the status quo).

> no population selection and reproduction algorithm

I never said there was. Darwin said it because he didn't know anything about genes, but that mistake was corrected by Dawkins.

> If gender equality and intellectual achievement don't produce children, then that isn't "darwinism selecting rationality out".

Why not?

> The people who believe in gender equality and intellectual achievement made the decision to not want more of themselves

That's the wrong way to look at it. Individuals are not the unit of reproduction. Genes are. Genes that build brains that want to have (and raise) children are more likely to propagate than genes that build brains that don't, all else being equial. So it is not rationality per se that is the problem -- rationality can provide a reproductive advantage because it lets you, for example, build technology. The problem is that non-rational brains can parasitically benefit from the phenotype of rational brains, at least for a while. But in the long run this is not a stable equilibrium.

Darwinism isn't a weakness of rationality, teleology has fine tuning problem, while darwinism is minimally fine tuned to work from scratch, which can be said to be optimal. Also darwinism doesn't select for reproductive fitness, it's only a proxy goal; true goal is survival, so you can produce more offspring only in a way compatible with true goal.

> Darwinism isn't a weakness of rationality

I didn't say it was. I said that the Rationalist community is not taking the implications of Darwinism into account when they choose where to focus their attention. This is what leads them to fixate on hostile AI and the MWI when what they should be worried about is the rise of MAGA. But not only is that not what they are worried about, many self-styled Rationalists are Trump supporters.

> If you are a gene, you can probably produce more offspring in an already-industrialized environment by making brains that lean more towards misogyny and sexual promiscuity than gender equality and intellectual achievement.

The fact that humans are intelligent at all and Enlightenment peoples currently dominate the world suggests otherwise.

> Enlightenment peoples currently dominate the world

Huh??? How do you figure? AFAICT the world is dominated by Donald Trump, Xi Jinping, and Vladimir Putin (if you reckon by power) or Christians and Muslims (if you reckon by population). None of these individuals or groups can be properly categorized as "Enlightenment peoples", certainly not with a capital E.

https://en.wikipedia.org/wiki/Age_of_Enlightenment

Western nations have dominated the world for centuries, in science, technology and cultural dominance. Russia is a bit player and has been for some time. Even if China were to ascend to equal prominence as the US, it would be an outlier in an otherwise clear trend.

> Western nations have dominated the world for centuries

I guess that depends on what you consider "the world". It makes no sense to even talk about the West dominating "the world" before 1492. The first truly global Western empire was Britain, but it was also the last. It was replaced by the U.S. but it was never really global. Even at the height of its power after WW2 the USSR was a credible rival. After the fall of the USSR in 1991 the U.S. was the sole undisputed superpower for a little while, but that came to an abrupt end on September 11, 2001 and the subsequent wars in Afghanistan and Iraq.

I think you are over-extrapolating the past into the future. The mindset and culture that produced U.S. hegemony in the 20th century seems to me to be mostly extinct. The U.S. was indeed ruled by rationalism (more or less) from the time of its founding through the mid-to-late 20th century, but there is precious little of that left today. Certainly the power structure in the U.S. today is actively hostile to rationalism, and I don't see a whole lot of rationalism in play in the opposition either.

Couple of point to that, mainly why should one have to follow darwinian evolution just because we are a product of that. It's similar to the natural law argument against homosexuality, that unnatural sex is wrong. The argument against that is natural biology does not inform what is good or what we should do.

I'm sure you would be able to predict what a rationalise will say when you ask them what future they prefer: one where we maximises for the number of humans or one with fewer humans but better lives

> why should one have to follow darwinian evolution

That depends on what you mean by "follow". You have to "follow" Darwinian evolution for the same reason you have to "follow", say, the law of gravity. That doesn't mean you can't build airplanes and spacecraft, but you still have to acknowledge and "follow" the law of gravity. You can't just glue feathers to your arms, jump off a cliff, and hope for the best. (Actually, rationalists aren't even gluing feathers to their arms. They are doing the equivalent of jumping off a cliff because they just don't believe gravity applies to them.)

[UPDATE]

> unnatural sex is wrong

The problem with that argument is that homosexuality is not unnatural. Many, many species have homosexual relations. Accounting for this is a little bit challenging, but the fact is undeniable.

https://en.wikipedia.org/wiki/Homosexual_behavior_in_animals

Pointing to other species can be futile since they also eat their young and are horrible partners :)

Well, yeah, the whole "against nature" argument is bogus to begin with. But I think a counter-argument is stronger if it can be made even while accepting the other side's premises.

An issue with this line of reasoning is that Darwinian evolution fails to accurately describe real evolutionary processes.

A counterexample is meiotic drive, where alleles disrupt the meiotic process in order to favour their own transmission, even if the alleles in question ultimately produce a less fit organism.

Whilst this is not an inherently positive observation, I think it does illustrate that the fatalistic picture you're painting here is incorrect. There's room for tentative optimism.

> Darwinian evolution fails to accurately describe real evolutionary processes.

That is not correct. Darwin did make a mistake, but it was not the fundamental dynamics of the process, but that he chose the wrong unit of selection. Darwin thought that selection selected for individuals or species when in fact it selects for genes. Richard Dawkins is the person who figured this out, but Darwin knew nothing about genes (OoS was published only three years after Gregor Mendel's work) so he still gets the credit nothwithsanding this mistake.

I don't think you've understood my point. I'm not suggesting that selection doesn't occur, but rather that selection is only part of the story. Have a look at gene flow or genetic drift if you want to know more, they're both examples of non-selective processes that influence the genetic makeup of a population.

> selection is only part of the story

Of course. Variation is the other part.

> gene flow or genetic drift

Those are two mechanisms by which variation occurs. This is not something Darwin got wrong, he just didn't have all the data. Genes were unknown in Darwin's time.

Which of selection or variation does meiotic drive fall under?

Clearly it isn't Darwinian selection, because it can favour transmission of genes which are not conducive to survival.

Clearly it isn't variation, because it tends to reduce the degree of genetic polymorphism within a population.

Meiotic drive is selection.

> it isn't Darwinian selection, because it can favour transmission of genes which are not conducive to survival

This is the one thing that Darwin got wrong: he thought that the unit of selection was the organism, but it's not. The unit of selection is the gene. Richard Dawkins is the one who figured this out. Meiotic drive is not Darinian selection, but it is selection. But Darwin could not possibly have known this because genes were unknown at the time.

But rationalism , if the modern sort, isn't supposed to be descriptive, it's supposed to be normative -- rationality lets you win.

The problem is not with rationalism, it's with Rationalism (with a capital R), the cult-like phenomenon that has grown up around rationalism that fetishizes things like Bayes's theorem, hostile AI, Roko's basilisk, and the MWI.

I used to think that massive, very long term droughts might cause serious instability, but I have since changed my mind. In highly developed nations, the amount of irrigation infrastructure built in the last 100 years is simple stunning. Plus, national agriculture research programmes are always researching how to use less water and grow the same amount of product. About drinking water: Rich places just build desalination plants. Sure, it is much more expensive than natural water sources (rivers, lakes, aquifers), but not expensive enough to cause political instability or serious economic harm. To be clear: Everything I wrote is from the perspective of highly developed nations. In middle income nations and below, droughts are incredibly challenging to overcome. The political and economic impacts can be enormous.

'America Is Using Up Its Groundwater Like There’s No Tomorrow' (https://waterwatch.org/america-is-using-up-its-groundwater-l...) to support that stunning irrigation infrastructure.

You should not interpret that historical success to imply future success as it depended on non-sustainable groundwater extraction.

Eg, https://en.wikipedia.org/wiki/Ogallala_Aquifer

> Many farmers in the Texas High Plains, which rely particularly on groundwater, are now turning away from irrigated agriculture as pumping costs have risen and as they have become aware of the hazards of overpumping.

> Sixty years of intensive farming using huge center-pivot irrigators has emptied parts of the High Plains Aquifer.

> as the water consumption efficiency of the center-pivot irrigator improved over the years, farmers chose to plant more intensively, irrigate more land, and grow thirstier crops rather than reduce water consumption--an example of the Jevons Paradox in practice

How will the Great Plains farmers get water once the remaining groundwater is too expensive to extract?

Salt Lake City cannot simply build desalination plants to fix its water problem.

I expect the bad experiences of Okies during the internal migration of the Dust Bowl will be replicated once the temporary (albeit century-long) relief of using fossil water is exhausted.

You raise an interesting point about inland population centers. My guess: Household daily water usage is a tiny fraction compared to Ag. As a result, at some point, I assume that states will slowly raise the cost of Ag water, until the most wasteful practices disappear. Example: They could move to drip irrigation that Israel uses, but it must at least 10x more expensive.

My limited understanding of the irrigation history in the US tells makes me extremely pessimistic about the likelihood of your prediction.

I think you only need to look at the water politics of the Great Salt Lake to see the difficulty.

Look at how little water use has changed during the last 25 years of the southwestern North American megadrought.

The US policy appears to be to pray for rain.

> Therefore, nothing can produce greater positive expected value than preventing existential risks—so working to reduce these risks becomes the highest priority.

Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are.

Suppose you clock "malicious AI" as a huge risk and then hamper AI, but it turns out the bigger risk is not doing space exploration, which AI would have accelerated, because something catastrophic yet already-inevitable is going to happen to the Earth in a few hundred years and if we're not sustainably multi-planetary by then it's all over.

The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

> Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are.

Rationalist community understands that very well. They even know how to put bounds on the unknowns and their own lack of information.

> The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

Right. Good thing they'd agree with you 100% on this.

The nature of unknowns is that you don't know them.

What's the probability of AI singularity? It has never happened before so you have no priors and any number you assign will be pure speculation.

Same is true about anything you're trying to forecast, by definition of it being in the future. And yet people have figured out how to make predictions more narrow than shrugging.

"And the general absolutist tone of the community. The people involved all seem very... Full of themselves ?"

>And yet people have figured out how to make predictions more narrow than shrugging

And?

>> It has never happened before

> Same is true about anything you're trying to forecast, by definition of it being in the future

There might be some flaws in this line of reasoning...

Yes, but making it more nuanced still doesn't change the point.

Singularity obviously never happened before, and if anyone bothered to read up on what they're talking about, they'd realize that no one is trying to predict what happens then, because the singularity is defined as the time when changes accelerate to such a degree that we have no baseline to make any predictions whatsoever.

So when people speculate on when that is, they're trying to forecast the point forecasting breaks; they do it by extrapolating from known examples and trends, to which we do have baselines.

Or, in short: we know how it is to ride an exponent, we just never rode one long enough to fall off of it; predicting singularity is predicting when the exponent gets steep enough we can't follow, which is not unlike predicting any other trend people do. Same methods and caveats apply.

"It is difficult to make predictions, especially about the future."

Most of the time we make predictions based on how similar events happened in the past. For completely novel situations it's close to impossible to make a prediction and reckless to base policy on such a prediction.

That's strictly true, but I feel like you're misunderstanding something. Most people aren't actually doing anything truly novel, hence very few people ever actually have to even attempt to predict things in this way.

But it was necessary at the beginning of flight and the flight to the moon would've never been possible either without a few talented people being able to make predictions about scenarios they knew little about.

There are just way too many people around nowadays, which is why most of us never get confronted with such novel topics and consequently we don't know how to reason about it

> They even know how to put bounds on the unknowns and their own lack of information.

No they don't. They think they can do this because they've accidentally reinvented the philosophy "logical positivism", which philosophers gave up on because it doesn't work. (This is similar to how they accidentally reinvented reconstructing arguments and called it "steelmanning".)

https://metarationality.com/probability-limitations

That's only one flaw in the theory.

There are others, such as the unproven, narcissistic and frankly unlikely-to-be-true assumption that humanity continuing to exist is a net positive in the long run.

A net positive for whom?

Everything. of which humanity is a miniscule part.

"net positive" requires a human being existing to judge.

> "net positive" requires a human being existing to judge.

This is effectively a religious belief you are espousing.

What do you mean? Religion seems like a total nonsequitur here. If anything I would argue the opposite: you would need religion or something like it for moral statements to have any meaning without the existence of sentient beings.

In what sense are people in those communities "quite smart"? Stupid is as stupid does. There are plenty of people who get good grades and score highly on standardized tests, but are in fact nothing but pontificating blowhards and useless wankers.

They're members of a religion which says that if you do math in your head the right way you'll be correct about everything, and so they think they're correct about everything.

They also secondarily believe everyone has an IQ which is their DBZ power level; they believe anything they see that has math in it, and IQ is math, so they believe anything they see about IQ. So if you avoid trying to find out your own IQ you can just believe it's really high and then you're going.

Unfortunately this lead them to the conclusion that computers have more IQ than them and so would automatically win any intellectual DBZ laser beam fight against them / enslave them / take over the world.

If only I could +1 this more than once! I have learned valuable things occasionally from people in the rationalist community but this overall lack of humility —and strangely blinkered view of humanities and important topics like say history of science relevant to “STEM”—ultimately turned me off to the movement as a whole. And I love science and math! It just shouldn’t belong to people with this (imo) childish model of people, IQ, etc.

According to rationalists, humans don't work together, so you can't add up their individual intelligence to get more intelligence. Meanwhile building a single giant super AI is technologically feasible, so they weigh the intelligence of a single person vs all AIs operating as a collective hivemind.

> According to rationalists, humans don't work together, so you can't add up their individual intelligence to get more intelligence.

An actual argument would be that intelligence doesn't work like that. Two people with IQ 100 cooperating together does not produce an IQ 200 solution.

There is the "wisdom of crowds". If a random member of a group is more than 50% likely to be correct, the average of the group is more likely to be correct than its members individually. But that has a few assumptions, for example that each member tries to figure out things independently (as opposed to everyone waiting for the highest-status member to express their opinion, and then agreeing with it -- in that case the entire group is only as smart as the highest-status member).

But you cannot leverage this to simply invite 1000 random people in your group and ask them to invent a Theory of Everything; because the assumption that each member is more than 50% likely to be correct does not apply in this case. So that is one of the limits of people working together.

(And this already conveniently ignores many other problems found in real life, such as conflict of interests, etc.)

I'm not familiar with any of these communities. Is there also a general bias towards one side between "the most important thing gets the *most* resources" and "the most important thing gets *all* the resources"? Or, in other words, the most important thing is the only important thing?

IMO it's fine to pick a favorite and devote extra resources to it. But that turns less fine when one also starts working to deprive everything else of any oxygen because it's not your favorite. (And I'm aware that this criticism applies to lots of communities.)

It's not the case. Effective altruists give to dozens of different causes, such as malaria prevention, environmentalism, animal welfare, and (perhaps most controversially) extinction risk. It can't tell you which root values to care about. It just asks you to consider whether the charity is impactful.

Even if an individual person chooses to direct all their donations to a single cause, there's no way to get everyone to donate to a single cause (nor is EA attempting to). Money gets spread around because people have different values.

It absolutely does take some money away from other causes, but only in the sense that all charities do: if you give a lot to one charity, you may have less money to give to others.

The general idea is that on the margin (in the economics sense), more resources should go to the most effective+neglected thing, and.the amount of resources I control is approximately zero in a global sense, so I personally should direct all of my personal giving to the highest impact thing.

And in their logic the highest impact is to donate money, take high paying jobs regardless of morality, and not focusing on any structural or root issues.

Yeah, the logic is basically "sure there are lots of structural or root issues, but I'm not confident I can make a substantial positive impact on those with the resources I have whereas I am confident that spending money to prevent people (mostly kids who would otherwise have survived to adulthood) from dying of malaria is a substantial positive impact at ~$5000 / life saved". I find that argument compelling, though I know many don't. Those many are free to focus on structural or root issues, or to try to make the case that addressing those issues is not just good, but better than reducing the impact of malaria.

I think most everyone can agree with this: Being 100% rigorous and rational, reasoning from first principles and completely discarding received wisdom is a great trait in a philosopher but a terrible trait in a policymaker. Because for the former, exploring ideas for the benefit of future generations is more important than whether they ultimately reach the right conclusion or not.

> Being 100% rigorous and rational, reasoning from first principles

It really annoys me when people say that those religious cultists do that.

They derive their bullshit from faulty, poorly thought out premises.

If you fuck up the very firsts calculations of the algorithm, it doesn't matter how rigorous all the subsequent steps are. There results are going to be all wrong.

The big problem is that human values are similar to neural network weights that can't be clearly defined into true/false axioms like "human life is inherently valuable" and "murder is wrong". The easiest axiom to break is something like "every human life is equally valuable", or even "every human life is born equal".

Technically "long-termism" should lead them straight to nihilism. Because, eventually, everything will end. One way or another. The odds are just 1. At some point, there are no more future humans. The number of humans are zero. Also, due to the nature of the infinite, any finite thing is essentially a rounding error and not worth concerning oneself with.

I get the feeling these people often want to seem smarter than they are, regardless of how smart they are. And they want to get money to ostensibly "consider these issues", but really they want money for nothing.

If they wanted to do right by the future masses, they should be looking to the things that are affecting us right now. But they treat those issues as if they'll work out in the wash.

> Technically "long-termism" should lead them straight to nihilism. Because, eventually, everything will end. One way or another. The odds are just 1. At some point, there are no more future humans. The number of humans are zero. Also, due to the nature of the infinite, any finite thing is essentially a rounding error and not worth concerning oneself with.

The current sums invested and donated in altruist causes are rounding errors themselves compared to GDPs of countries, so the revealed preferences of those investing and donating to altruist causes is to care about the future and the present also.

Are you saying that they should give a greater preference to help those who already exist rather than those who may exist in the future?

I see a lot of Peter Singer’s ideas in modern “effective” altruism, but I get the sense from your comment that you don’t think that they have good reasons for doing what they do, or that their reason leads them to support well-meaning but ineffective solutions. I am trying to understand your position without misrepresenting your point or goals. Are you naysaying or do you have an alternative?

https://en.wikipedia.org/wiki/Peter_Singer

I think it's essentially a grift. An excuse to do nothing while looking like you care and reaping rewards.

If they wanted to help, they should be focused on the now. Global poverty, climate change, despotic world leaders. They should be aligning themselves against such things.

But instead what we see is essentially not that. Effective altruism is a lot like the Democratic People's Republic of Korea, a bit of a misnomer.

Agreed.

To be pedantic, DPRK is run via the will of the people to a degree comparable to any country. A bigger misnomer is the west calling liberal “democracy”, just democracy.

Even if I understand what you mean, DPRK is very far away from democratic countries. It's not comparable at all.

My point is that most people who see that think a country like America or Europe are democracies and that is partly why it would be a misnomer. When it is arguable that DPRK state is more a will of the people than the west.

The elite letting the people choose between a few candidates is not a democracy. There are no “democratic” countries in that way.

“ When it is arguable that DPRK state is more a will of the people than the west.”

Ba da Ching

What do people like you use to prove the DPRK is auth, a dictatorship, and so bad while so smugly believing they have “freedom” and “democracy” better than the Global South? Western state depts and western state directed media manufacturing consent - ba da ching

Don’t forget — the enemies of the west are all bad and evil. And the west is free and fair and democratic and not a scourge on the rest of the world.

Because when doctors go there to cure the blind from easily treatable malnutrition the people drop to the floor in fear praising the moon lord leader.

> When it is arguable that DPRK state is more a will of the people than the west.

It's not arguable, it's simply wrong. But to understand that you would have to understand much more about the DPRK.

> The elite letting the people choose between a few candidates is not a democracy.

It is, but more importantly, your framing is wrong. Every democracy has several levels of democratic institutions (local, state, federal etc.), there are often 'surprise' election winners; there are also non-elective ways for non-elite people to influence policy. DPRK has none of this.

> But to understand that you would have to understand much more about the DPRK.

I have never come across some one like you who knows about DPRK outside of what western state depts say and of course they swallow it all up. But people do sometimes lie and say they know so much like you right now

Now you did come across someone like that, so you should update your assumptions. But more importantly study more and don't be a tankie.

using tankie unironically. I refer to tankie Dr MLK saying call it democracy or call it democratic socialism.

They have aligned themselves in favor of global poverty, climate change and despotic world leaders.

A lot of them argue that poor countries essentially don't matter. Climate change is not an extinction event and there should an authoritarian world government to prevent nuclear conflict to minimize the risk of nuclear extinction.

>In his dissertation On the Overwhelming Importance of Shaping the Far Future (2013), supposedly “one of the best texts on existential risks,”[9] Nicholas Beckstead meditates on the “ripple effects” a human life might have for future generations and concludes “that saving a life in a rich country is substantially more important than saving a life in a poor country” due to the higher level of innovation and economic productivity attained in these countries.[10]

https://umbau.hfg-karlsruhe.de/posts/philosophy-against-the-...

Meanwhile, on an actual EA website: https://www.givewell.org/charities/top-charities

The site you reference quotes Beckstead out of context, and either reading the context or looking at what he spent the next decade of his life working on would make it clear that he thinks the marginal dollar is better spent on saving lives in poor countries than rich ones. He well understands that his "other things being equal" in his dissertation essentially never holds in practice, and was writing for a philosophy audience where this kind of hypothetical is expected.

> But they treat those issues as if they'll work out in the wash.

there's something pathologically, virally sick about wealth accumulation's primary function being the further accumulation of wealth. for a movement rooted in "rationalism," EA seems pretty irrationally focused on excusing antisocial behavior.

> wealth accumulation's primary function being the further accumulation of wealth

Then we are lucky that EA promotes giving more to charity as the primary function of accumulation of wealth.

> EA seems pretty irrationally focused on excusing antisocial behavior.

Is a guy getting a well-paid job at Microsoft and donating half of his salary to African charities really your best example of antisocial behavior?

> Then we are lucky that EA promotes giving more to charity as the primary function of accumulation of wealth.

no, we are not lucky. EA-good-because-charity-good is a brain-pretzel way of lobbying against equitable taxation.

> Is a guy getting a well-paid job at Microsoft and donating half of his salary to African charities really your best example of antisocial behavior?

you're inventing a competitive debate regarding a hypothetical "best example of antisocial behavior". i didn't target anyone specifically with any part of my post.

Well, I base my opinion about EA on specific people I happen to know, such as https://www.jefftk.com/news/ea , but of course that shouldn't stop anyone from making up edgy interpretations and posting them as a fact. Because what is the point of discussing actual effective altruists when imaginary villains are a much more interesting topic.

What it promotes and what happens are two different things.

“The purpose of a system is what it does.”

The other weird direction it leads is space travel.

If you assume we eventually figure out long distance space travel and humanity spreads across the galaxy, there could in the future be quadrillions of people, growing at some kind of exponential rate. So accelerating the space race by even an hour is equivalent to bringing billions of new souls into existence.

I don't see how bringing new souls (whatever those are) into existence should naturally qualify as a good thing?

Perhaps you're arguing as an illustration of the way this group of people think, in which case I understand your point.

You can make arguments for it starting with "killing billions of people would definitely be bad" or "some fraction of those people will likely share my genes, which I have a biological drive to pass on".

It encodes a slight bias towards human existence being a positive thing for us humans, but I don't think it's the shakiest part of that reasoning.

Nitpick: it bottoms out at quadratic growth in the limit, not exponential.

>People in these communities are generally quite smart, and it’s seductive to reason in a purely logical, deductive way. There is real value in thinking rigorously and in making sure you’re not beholden to commonly held beliefs. But, like you said, reality is complex, and it’s really hard to pick initial premises that capture everything relevant. The insane conclusions they get to could be avoided by re-checking & revising premises, especially when the argument is going in a direction that clashes with history, real-world experience, or basic common sense.

They don't even do this.

If you're reasoning in purely logical and deductive way - it's blatantly obvious that living beings experience way more pain and suffering, than pleasure and joy. If you do the math, humanity getting wiped out in effect is the best thing that could happen.

Which is why accelerationism ignoring all the AGI risks is correct strategy presuming the AGI will either wipe us out (good outcome) or provide technologies that improve the human condition and reduce suffering (good outcome).

Logical and deductive reasoning based on completely baseless and obviously incorrect premises is flat out idiotic.

You can't deprive non-existent people out of anything.

And if you do, I hope you're ready for purely logical, deductive follow up - every droplet of sperm is sacred and should be used to impregnate.

For anyone who takes Effective Altruism or Rationalism seriously I strongly recommend reading "You Are Not a Gadget" It was written more than ten years ago and was so prescient about the issues of social media and also contained one of the most devastating problems with EA, the idea of a circle of empathy.

You don't have to agree with any of this. I am not defending every idea the author has. But I recommend that book.

Thanks for the reco.

What's the seductive-but-wrong part of EA? As far as I can tell, the vast majority of the opposition to it boils down to "Maybe you shouldn't donate money to pet shelters when people are dying of preventable diseases" vs "But donating money to pet shelters feels better!"

Yes, but they’re smug about it, which means they’re wrong.

Of course it sounds ridiculous when you spell it out this way.

Or something like focusing on high paying job and donating money does absolutely nothing to solve any root or structural problems which defeats the supposed purpose of caring about future people or being effective altruist in the first place

Of course the way your comment is written makes criticism sound silly.

It might be the parts that lead a person to commit large scale fraud with the idea that the good they can do with the stolen money outweighs all the negatives. Or, at least, that’s the popular idea of what happened to Sam Bankman-Fried. I have no idea what was actually going through that man’s mind.

In any case, EA smells strongly of “the ends justify the means” which most popular moral philosophies reject with strong arguments. One which resonates with me is that there are no “ends.” The path itself is the goal.

> the ends justify the means” which most popular moral philosophies reject with strong arguments.

This is a false statement. Our entire modern world is built on the basis of the ends justify the means. Every time money is spent on long term infrastructure vs giving poor kids food right now, every time a war is fought, every time a doctor triages injuries at a disaster.

I don't think it's useful to conflate "the ends justify the means" with "cost-benefit analysis". You sometimes use the latter to justify certain means, but you don't have to, that's why they're different. When you believe that the ends justify the means, you can also just give no consideration at all to the ethics of the means. No doctor triaging patients would ever shoot a patient in the head so he could move onto one they thought was more important. Yes they might let patients die, but that's different than actively killing them.

In the framework of modern civilian medicine sure.

I'm sure exactly what you described was done plenty of times in ww1 and similar around that era, and seen as perfectly moral and rational.

I've noticed this with a lot of radical streamers on both sides. They don't care about principles, they care about "winning" by any means necessary.

Winning at things that align with your principle is a principle. If you don't care about principles, you don't care about what you're winning at, thereby making every victory hollow and meaningless. That is how you turn into a loser at everything you do.

> there are no “ends.” The path itself is the goal.

How does this apply to actual charity? "Curing malaria is not the goal. Our experiences during voluntourism are the true goal."

It's not all bad, some conclusions are ok. But it also has ideas like "don't donate to charity, it's better to invest that money in, like, an oil fund, and grow it 1000x, and then you can donate so much more! Bill Gates has done much more for humanity than some Red Cross doctor!". Which is basically just a way to make yourself feel good about becoming richer, much like "prosperity gospel" would for the religious.

This is not a commonly-held position in EA.

That hasn't defined EA for at least 10 years or so. EA started with "you should donate to NGOs distributing mosquito nets instead of the local pet shelter".

It then moved into "you should work a soulless investment banking job so you can give more".

More recently it was "you should excise all expensive fun things from your life, and give 100% of your disposable income to a weird poly sex cult and/or their fraudulent paper hedge fund because they're smarter than you."

This is what some EAs believe, I don't think there was ever a broad consensus on those latter claims. As such, it doesn't seem like a fair criticism of EA.

You can’t paint a broad brush of a whole movement, but it’s true if the leadership of the EA organization. Once they went all-in on “AI x-risk”, there ceased being a meaningful difference between them and the fringe of the LW ratsphere.

Which organization?

https://forum.effectivealtruism.org/posts/f6kg8T2Lp6rDqxWwG/...

The people who run the forum you linked to.

That same link puts AI risk under the "far future" category, basically the same category as "threats to global food security" and asteroid impact risks. What's unreasonable about that?

> That hasn't defined EA for at least 10 years or so.

Meanwhile, on an actual EA website: https://www.givewell.org/charities/top-charities

* Medicine to prevent malaria

* Nets to prevent malaria

* Supplements to prevent vitamin A deficiency

* Cash incentives for routine childhood vaccines

GiveWell is an example of the short-termist end of EA. At the long-termist end people pay their friends to fantasize about Skynet at 'independent research institutes' like MIRI and Apollo Research. At the "trendy way to get rich people to donate" end you get buying a retreat center in Berkley, a stately home in England, and a castle in Czechia so Effective Altruists can relax and network.

Its important to know which type of EA organization you are supporting before you donate, because the movement includes all three.

I assume that GiveWell is the most popular of them. I mean, if you donate to MIRI, it is because you know about MIRI and because you specifically believe in their cause. But if you are just "hey, I have some money I want to donate, show me a list of effective charities", then GiveWell is that list.

(And I assume that GiveWell top charities receive orders of magnitude more money, but I haven't actually checked the numbers.)

Even GiveWell partnered with the long-termist/hypothetical risk type of EA by funding something called Open Philanthropy. And there are EA organizations which talk about "animal welfare" and mean "what if we replaced the biosphere with something where nothing with a spinal cord ever gets eaten?" So you can't trust "if it calls itself EA, it must be highly efficient at turning donations into measurable good." EA orgs have literally hired personal assistants and bought stately homes for the use of the people running the orgs!

Pet shelters is just the pretty facade. The same ideology would have you step over homeless elderly ladies and buy mosquito nets instead, because the elderly lady cannot have children so mosquito nets maximize the number of future humans more efficiently.

EA always rubbed me the wrong way.

(1) The kind of Gatesian solutions they like to fund like mosquito nets are part of the problem, not part of the solution as I see it. If things are going to get better in Africa, it will be because Africans grow their economy and pay taxes and their governments can provide the services that they want. Expecting NGOs to do everything for them is the same kind of neoliberal thinking that has rotted state capacity in the core and set us up for a political crisis.

(2) It is one thing to do something wrong, realize it was a mistake, and then make amends. It's another thing to do plan to do something wrong and to try to offset it somehow. Many of the high paying jobs that EA wants young people to enter are "part of the problem" when it comes to declining stage capacity, legitimation crisis, and not dealing with immediate problems -- like the fact that one of these days there's going to be a heat wave that is a mass causality event.

Furthermore

(3) Time discounting is a central part of economic planning

https://en.wikipedia.org/wiki/Social_discount_rate

It is controversial as hell, but one of the many things the Soviet Union got wrong before the 1980s was planning with a discount rate of zero, which led to many economically and ecologically harmful projects. If you seriously think it should be zero you should also be considering whether anybody should work in the finance industry at all or if we should have dropped a hydrogen bomb on Exxon's headquarters yesterday. At some point speculations about the future are just speculation. When it comes to the nuclear waste issue, for instance, I don't think we have any idea what state people are going to be in 20,000 years. They might be really pissed that buried spent nuclear fuel some place they can't get at it. Even the plan to burn plutonium completely in fast breeder reactors has an air of unreality about it, even though it happens on a relatively short 1000 year timescale we can't be sure at all that anyone will be around to finish the job.

(4) If you are looking for low-probability events to worry about I think you could find a lot of them. If it was really a movement of free thinkers they'd be concerned about 4,000 horsemen of the apocalypse, not the 4 or so that they are allowed to talk about -- but talk about a bunch of people who'll cancel you if you "think different". Somehow climate change and legitimation crisis just get... ignored.

(5) Although it is run by people who say they are militant atheists, the movement has all the trappings of a religion, not least "The Singularity" was talked about by Jesuit Priest Teilhard de Chardin long before sci-fi writer Vernor Vinge used it as the hinge of a mystery novel.

(3) "Controversial" is a weasel word AFAIC :)

The difficulty is in deriving any useful utility function from prices (even via preferences :), and as you know, econs can't rid themselves of that particular intrusive thought

https://mitsloan.mit.edu/sites/default/files/inline-files/So...

E: know any econs taking Habermas seriously ? Not a rhetorical q:

http://ecoport.org/storedReference/558800.pdf

What makes it really bad is that you can't add different people's utility functions, or for that matter, multiply the utility function of some imagined galactic citizen by some astronomical amount. The question of "what distribution of wealth maximizes welfare" [1] is unanswerable in that framework and we're left with the Randian maxim that any transaction freely entered into is "fair" because nobody would enter into it if it didn't increase their utility function.

[1] Though you might come to the conclusion that greeder people should have the money because they like it more

Comic sophism ain't gonna make hypernormies reconsider lol

(Aside from the semi-tragic one to consider additive dilogarithms..)

One actionable (utility agnostic) suggestion: study the measureable consequences of (quantifiable) policy on carbon pricing, because this is already quite close to the uncontroversial bits

E.g. precisely how inflationary can we make carbon credits?

E: by "uncontroversial", I meant amongst the Orthodox econs, so not Graeber & sympathetic heterodox.

> When it comes to the nuclear waste issue, for instance ...

Nuclear waste issues are 99.9% present-day political/ideological. Huge portions of the Earth are uninhabitable due to climate and/or geology. Lead, mercury, arsenic, and other naturally-occurring poisons contaminate large areas. Volcanoes spew CO2 and toxic gasses by the megaton.

Vs. when is the last time you heard someone get excited over toxic waste left behind by the Roman Empire?

I agree with your lead (the issue is 99% political) but man does that last bit demand a rebuttal. Waste left behind by the roman empire isn't even remotely comparable to long term radioactive material. I suggest having a look through the wikipedia list of orphan source incidents to get an idea of what happens when people unknowingly come across radioactive material. https://en.wikipedia.org/wiki/List_of_orphan_source_incident...

I think of the widespread heavy metal contamination in China which existed even before the modern age of manufacturing. Some might be naturally occurring but you had people like this guy

https://en.wikipedia.org/wiki/Mausoleum_of_Qin_Shi_Huang

That's an interesting list - but the injury & death rates are several orders of magnitude below even lightning strikes. https://en.wikipedia.org/wiki/Lightning_injury#Epidemiology

Also, PaulHoule's original comment said "in 20,000 years". Cobalt 60 (for example) has a half-life of 5 1/4 years - so there really won't be any of it left by then.

What's the rate conditioned on being near an incident though? And these are small, isolated incidents. How does what we see extrapolate to large scale nuclear waste storage, a state that failed a few hundred years ago, and someone inadvertently digging it up?

No one is talking about stuffing cobalt 60 in yucca mountain (at least as far as I know).

Compared to abandoned/forgotten mines (that eventually cave in) and mega-scale chemical waste dumps/sites/spills, nuclear waste sites - especially ones that'll still be seriously dangerous centuries or millennia from now - are profoundly rare.

And the tech to detect that you're digging into radioactive stuff is far simpler than the tech to detect that you're digging into some sort of chemical waste, or a failing old mine or tunnel.

If millennia-in-the-future humans care all that much about what we did with our nuclear waste, it'll either be political/ideological, or (as PaulHoule suggested) just one more "they didn't leave it somewhere really convenient for us" deal.

>when is the last time you heard someone get excited over toxic waste left behind by the Roman Empire?

For archeologists, pretty much every time.

> If things are going to get better in Africa, it will be because Africans grow their economy and pay taxes and their governments can provide the services that they want

The problem with this argument is that the path to achieve this is unclear, and everyone who has tried has failed. In the absence of a clear path, it seems rational to set aside lofty ideals and do whatever good you can now.

> If you are looking for low-probability events to worry about I think you could find a lot of them.

Name one that hasn't already been considered that would be a serious threat to modern technological civilization.

> Although it is run by people who say they are militant atheists, the movement has all the trappings of a religion

Supposing you're correct this thought is incomplete. Take it to its logical conclusion: a religion centered around open rational debate is bad because...?

> "The Singularity" was talked about by Jesuit Priest Teilhard de Chardin long before sci-fi writer Vernor Vinge used it as the hinge of a mystery novel.

Similarly, Big Bang was talked about by Catholic priest Georges Lemaître, and Bayes' Theorem was invented by Presbyterian minister Thomas Bayes. Does that prove anything beyond the fact that there are many smart religious people?

> then even small probabilities of catastrophic events wiping out humanity yield enormous negative expected value. Therefore, nothing can produce greater positive expected value than preventing existential risks—so working to reduce these risks becomes the highest priority.

This is the logic of someone who has failed to comprehend the core ideas of Calculus 101. You cannot use intuitive reasoning when it comes to infinite sums of numbers with extremely large uncertainties. All that results is making a fool out of yourself.

This summarizes what irks me about the community.

They use technical terms (eg expected value, KL divergence) in their verbal reasoning only to sound rational, but don’t ever mean to use those terms technically.

This tracks based on my limited contact with LessWrong during the whole Roko's Basilisk thing.

I quickly lost interest in Roko's Basilisk, but that is what brought me in the door and started me looking around the discussions. At first, it was quite seductive. There was a strange fearlessness there, a willingness to say and admit some things about humanity, our limitations and how we tend to think that other great thinkers maybe danced around in the past. After awhile it became clear that while there were a select few individuals who had found some balance between purely rational thinking and how reality actually works, most of the rest had their heads so far up their asses that they'd fart and call it a cool breeze. Reminded me of my brief obsession with Game Theory and realizing that even it's creators knew it's utility was not quite as advertised to the layman (as in it would not really help you predict or plan for anything at all, just model how decisions might be made).

Peardon the "it's" autocorrect, I can't seem to edit on mobile (Harmonic)

Just musing out loud. "Utility" is like, for

Physics postgrads: "gauge"

Physics undergrads: "wavefunction"

Grade schoolers: "temperature"

These concepts definitely useful for the hw sets, no understanding needed (or expected!)

It seems self-evident to me that epistemic humility _requires_ temporal discounting. We should not be confident we can predict the future well enough to integrate utility centuries forward.

I read the whole tree of responses under this comment and I could only convince myself that when people have no arguments they try to make you look bad.

Most of criticisms are just "But they think they are better than us !" and the rest is "But sometimes they are wrong !"

I don't know about the community and couldn't care less but their writings have brought me some almost life saving fresh air in how to think about the world. It is very sad to me to read so many falsely elaborate responses from supposedly intelligent people having their ego hurt but in the end it reminds me why I like rationalists and I don't like most people.

Feels like "they are wrong and smug" is enough reason to dislike the movement

Bashir: Even when they're neither?

Garak: Especially when they're neither.

The comment I replied to conceded wrongness and smugness but is still somehow confused why otherwise intelligent ppl dislike the movement. I was hoping to clear it up for them

Extra points for that comment's author implying that people who don't like the wrong and smug movement are unintelligent and protecting their egos, thus personally proving its smugness

I only conceded it insofar as everyone can be wrong sometimes, but at least those people seem to have a protocol to deal with it.Most people don't, and are fine with it. I stand on the side of those who care and are trying.

As for smugness, it is subjective. Are those people smug ? Or are they talking passionately about some issue with the confidence of someone who feel what they are talking about and are expecting for it to resonate ? It's the eye of the beholder I guess.

For example what you call my smugness is what I would a slightly depressed attitude fueled by the fact that it's sometimes hard to relate to other people feelings and behavior.

Here's a theory of what's happening, both with you here in this comment section and with the rationalists in general.

Humans are generally better at perceiving threats than they are at putting those threats into words. When something seems "dangerous" abstractly, they will come up with words for why---but those words don't necessarily reflect the actual threat, because the threat might be hard to describe. Nevertheless the valence of their response reflects their actual emotion on the subject.

In this case: the rationalist philosophy basically creeps people out. There is something "insidious" about it. And this is not a delusion on the part of the people judging them: it really does threaten them, and likely for good reason. The explanation is something like "we extrapolate from the way that rationalists think and realize that their philosophy leads to dangerous conclusions." Some of these conclusions have already been made by the rationalists---like valuing people far away abstractly over people next door, by trying to quantify suffering and altruism like a math problem (or to place moral weight on animals over humans, or people in the future over people today). Other conclusions are just implied, waiting to be made later. But the human mind detects them anyway as implications of the way of thinking, and reacts accordingly: thinking like this is dangerous and should be argued against.

This extrapolation is hard to put into words, so everyone who tries to express their discomfort misses the target somewhat, and then, if you are the sort of person who only takes things literally, it sounds like they are all just attacking someone out of judgment or bitterness or something instead of for real reasons. But I can't emphasize this enough: their emotions are real, they're just failing to put them into words effectively. It's a skill issue. You will understand what's happening better if you understand that this is what's going on and then try to take their emotions seriously even if they are not communicating them very well.

So that's what's going on here. But I think I can also do a decent job of describing the actual problem that people have with the rationalist mindset. It's something like this:

Humans have an innate moral intuition that "personal" morality, the kind that takes care of themselves and their family and friends and community, is supposed to be sacrosanct: people are supposed to both practice it and protect the necessity of practicing it. We simply can't trust the world to be a safe place if people don't think of looking out for the people around them as a fundamental moral duty. And once those people are safe, protecting more people, such as a tribe or a nation or all of humanity or all of the planet, becomes permissible.

Sometimes people don't or can't practice this protection for various reasons, and that's morally fine, because it's a local problem that can be solved locally. But it's very insidious to turn around and justify not practicing it as a better way to live: "actually it's better not to behave morally; it's better to allocate resources to people far away; it's better to dedicate ourselves to fighting nebulous threats like AI safety or other X-risks instead of our neighbors; or, it's better to protect animals than people, because there are more of them". It's fine to work on important far-away problems once local problems are solved, if that's what you want. But it can't take priority, regardless of how the math works out. To work on global numbers-game problems instead of local problems, and to justify that with arguments, and to try to convince other people to also do that---that's dangerous as hell. It proves too much: it argues that humans at large ought to dismantle their personal moralities in favor of processing the world like a paperclip-maximizing robot. And that is exactly as dangerous as a paperclip-maximizing robot is. Just at a slower timescale.

(No surprise that this movement is popular among social outcasts, for whom local morality is going to feel less important, and (I suspect) autistic people, who probably experience less direct moral empathy for the people around them, as well as to the economically-insulated well-to-do tech-nerd types who are less likely to be directly exposed to suffering in their immediate communities.)

Ironically paperclip-maximizing-robots are exactly the thing that the rationalists are so worried about. They are a group of people who missed, and then disavowed, and now advocate disavowing, this "personal" morality, and unsurprisingly they view the world in a lens that doesn't include it, which means mostly being worried about problems of the same sort. But it provokes a strong negative reaction from everyone who thinks about the world in terms of that personal duty to safety, because that is the foundation of all morality, and is utterly essential to preserve, because it makes sure that whatever else you are doing doesn't go awry.

(edit: let me add that your aversion to the criticisms of rationalists is not unreasonable either. Given that you're parsing the criticisms as unreasonable, which they likely are (because of the skill issue), what you're seeing is a movement with value that seems to be being unfairly attacked. And you're right, the value is actually there! But the ultimate goal here is a synthesis: to get the value of the rationalist movement but to synthesize it with the recognition of the red flags that it sets off. Ignoring either side, the value or the critique, is ultimately counterproductive: the right goal is to synthesize both into a productive middle ground. (This is the arc of philosophy; it's what philosophy is. Not re-reading Plato.) The rationalists are probably morally correct in being motivated to highly-scaling actions e.g. the purview of "Effective Altruism". They are getting attacked for what they're discarding to do that, not for caring about it in the first place.)

I finally gave in and created an account because of your comment. It's beautifully put. I would only perhaps add that, to me, the neo-rationalist thing looks the most similar to things that don't work yet attract hardcore "true believers". It's a pattern repeated through the ages, perhaps most intimately for me in the seemingly endless parades of computer system redesigns: software, hardware, or both. Randomly one might pick "the new and exciting Digg", the Itanium, and the Metaverse as fairly modern examples.

There is something about a particular "narrowband" signaling approach, where a certain kind of purity is sought, with an expectation that, given enough explaining, you will finally get it, become enlightened, and convert to the ranks. A more "wideband" approach would at least admit observations like yours do exist and must be comprehensively addressed to the satisfaction of those who hold such beliefs vs to the satisfaction of those merely "stooping" to address them (again in the hopes they'll just see the light so everyone can get back to narrowband-ville).

(Thank you) I agree, although I do think that the rationalists and EAs are way better than most of the other narrowband groups, as you call them, out there, such as the Metaverse or Crypto people. The rationalists are at least mostly legitimately motivated by morality and not just by a "blow it all up and replace it with something we control" philosophy (which I have come to believe is the belief-set that only a person who is convinced that they are truly powerless comes to). I see the rationalists as failing due to a skill issue as well: because they have so-defined themselves by their rationalism, they have trouble understanding the things in the world that they don't have a good rational understanding of, such as morality. They are too invested in words and truth and correctness to understand that there can be a lot of emotional truth encoded in logical falsehood.

edit: oh, also, I think that a good part of people's aversion to the rationalists is just a reaction to the narrowband quality itself, not to the content. People are well-aware of the sorts of things that narrowband self-justifying philosophies lead to, from countless examples, whether it's at the personal level (an unaccountable schoolteacher) or societal (a genocidal movement). We don't trust a group unless they specifically demonstrate non-narrowbandedness, which means being collectively willing to change their behavior in ways that don't make sense to them. Any movement that co-opts the idea of what is morally justifiable---who says that e.g. rationality is what produces truth and things that run counter to it do not---is inherently frightening.

They aren’t motivated by morality. They just are more moral relative to the niches you referred to.

Any group that focuses on their own goals of high paying jobs regardless of the morality of those jobs or how they contribute to the structural issues of society is not that good. Then donating money while otherwise being okay with the status quo —- not touching anything systemic in such an unjust world but supposedly focusing on morality is laughable.

I see it as moral action with a skill issue, which is how I see a lot of bad things in the world. People are very good at missing the forest for the trees.

In their defense they do try to do the calculations: is a high-paying job okay on net if you give most of the money away? depends on the job; depends on if somebody else would have it if you didn't; etc. Not that there is a rigorous way to do them but it's very much a group that does try.

> Ironically paperclip-maximizing-robots are exactly the thing that the rationalists are so worried about. They are a group of people who missed, and then disavowed, and now advocate disavowing, this "personal" morality, and unsurprisingly they view the world in a lens that doesn't include personality morality, which means mostly being worried about problems of the same sort. But it provokes a strong negative reaction from everyone who thinks about the world in terms of that personal duty to safety.

I head not read any rationalist writing in a long time (and I didn't know about Scott's proximity), but the whole time I was reqding the article I was thinking the same thing you just wrote... "why are they afraid of AI, i.e. the ultimate rationalist taking over the world", maybe something deep inside of them has the same reaction to their own theories as you so eloquently put above.

I don't read these rationalist essays either, but you don't need to be a deep thinker to understand why any rational person would be afraid of AI and the singularity.

The AI will do what its programmed to do, but its programmers morality may not match my own. What more scary is that it may be developed with the morality of a corporation rather than a person. (That is to say, no morals at all.)

I think its perfectly justifiable to be scared of a very powerful being with no morals stomping around!

Those corporations are already superhuman entities with morals that don’t match ours. They do cause a lot of problems. Maybe it’s better to figure out how to fix that real, current problem rather than hypothetical future ones.

Yeah, the AI will be a tool of a corp. So in effect we are talking about limiting corporate power.

> the AI will be a tool of a corp.

Up to a certain level. Behind that level, the corp will be a tool of the AI. And even further that way, the corp will no longer be necessary.

This parallel has been drawn. Charlie Stross [0] in particular thinks the main difference is that pre-digital AIs are capable of behaving much faster, so that other entities (countries, lawmakers, …) have time to react to them.

[0]: https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...

Similar claims can be made about any structure of humans that exhibits gestalt intelligence, e.g. nations, stock markets, etc.

It's quite likely I got it from that!

What creeps me out is that I have no idea of their theory of power: How will they achieve their aims?

Maybe they want to do it in a way I’d consider just: By exercising their rights as individuals in their personal domains and effectively airing their arguments in the public sphere to win elections.

But my intuition is they think democracy and personal rights of the non-elect are part of the problem to rationalize around and over.

Would genuinely love to read some Rationalist discourse on this question.

Why is "theory of power" a necessary framing? "Theory of power" seems to be only a popular idea in "online left progressive" circles which are based on pop readings of post-structuralist continental philosophy. There's plenty of other schools of thought out there and there have been many criticisms of the idea of "theory of power" altogether.

Reading critiques of Hegel is a great starting point for this reading.

Whether you accept it or not though, there's lots of non-rationalist schools that reject the need for a "theory of power".

I didn’t mean the term in anything but a colloquial sense: They believe certain outcomes are right and proper. Such outcomes don’t typically manifest themselves out of good intentions. What’s the plan?

When Curtis Yarvin is at least in your orbit, these should not be surprising questions to get.

you've latched on to some academic sense of the term but I understood exactly what the person you're replying to meant. They mean: are these people you would respect and trust to be in charge of things? What would they do? And a lot of people's vibe-check says it's a big unknown question mark, or worse, which is not something you can put faith in.

It's hard to argue with vibes which is what makes today's culture war based politics so difficult to weigh in on conclusively. It's all just vibes after all. One person's vibes can be cap to someone else.

There's a reason technofeudalists like Yarvin are popular in rationalist adjacent circles.

No offense, but this way of thinking is the domain of comic book supervillains. "I must destroy the world in order to save it." Morality is only holding me back from maximizing the value of the human race 1,000 or 1,000,000 years from now type nonsense.

This sort of reasoning sounds great from 1000 feet up, but the longer you do it the closer you get to "I need to kill nearly all current humans to eliminate genetic diseases and control global warming and institute an absolute global rationalist dictatorship to prevent wars or humanity is doomed over the long run".

Or you get people who are working in a near panic to bring about godlike AI because they think that once the AI singularity happens the new AI God will look back in history and kill anybody who didn't work their hardest to bring it into existence because they assume an infinite mind will contain infinite cruelty.

but it will also contain infinite love.

the correct answer is probably that it could contain infinite of any of these things, but you don't know which one it's going to be a priori, and you get one shot to be right.

Holy heck this is so well put and does the exact thing where it puts into words how I feel which is hard for me to do myself.

> The explanation is something like "we extrapolate from the way that rationalists think and realize that their philosophy leads to dangerous conclusions."

I really like the depth of analysis in your comment, but I think there's one important element missing, which is that this is not an individual decision but a group heuristic to which individuals are then sensitized. Individuals don't typically go so far as to (consciously or unconsciously) extrapolate others' logic forward to decide that it's dangerous. Instead, people get creeped out when other people don't adhere to social patterns and principles that are normalized as safe in their culture, because the consequences are unknown and therefore potentially dangerous; or when they do adhere to patterns that are culturally believed to be dangerous. This can be used successfully to identify things that are really dangerous, but also has a high false positive rate (people with disabilities, gender identities, or physical characteristics that are not common or accepted within the beholder's culture can all trigger this, despite not posing any immediate/inherent threat) as well as a high false negative rate (many serial killers are noted to have been very charismatic, because they put effort into studying how to behave to not trigger this instinct). When we speak of something being normalized, we're talking about it becoming accepted by the mainstream so that it no longer triggers the ‘creepy’ response in the majority of individuals. As far as I can tell, the social conservative basically believes that the set of normalized things has been carefully evolved over many generations, and therefore should be maintained (or at least modified only very cautiously) even if we don't understand why they are as they are, while the social liberal believes that we the current generation are capable of making informed judgements about which things are and aren't harmful to a degree that we can (and therefore should) continuously iterate on that set to approach an ideal goal state in which it contains only things that are factually known to be harmful.

As an interesting aside, the ‘creepy’ emotion, (at least IIRC in women) is triggered not by obviously dangerous situations but by ambiguously dangerous situations, i.e. ones that don't obviously match the pattern of known safe or unsafe situations.

> Sometimes people don't or can't practice this protection for various reasons, and that's fine; it's a local problem that can be solved locally. But it's very insidious to turn around and justify not practicing it: "actually it's better not to behave morally; it's better to allocate resources to people far away; it's better to dedicate ourselves to fighting nebulous threats like AI safety or other X-risks instead of our neighbors".

The problem with the ‘us before them’ approach is that if two neighbourhoods each prioritize their local neighbourhood over the remote neighbourhood and compete (or go to war) to better their own neighbourhood at the cost of the other, generally both neighbourhoods are left worse off than they started, at least in the short term: both groups trying to make locally optimal choices leads (without further constraints) to globally highly suboptimal outcomes. In recognition of this a lot of people, not just capital-R Rationalists, now believe that at least in the abstract we should really be trying to optimize for global outcomes.

Whether anybody realistically has the computational ability to do so effectively is a different question, of course. Certainly I personally think the future-discounting ‘bias’ is a heuristic used to acknowledge the inherent uncertainty of any future outcome we might be trying to assign moral weight to, and should be accorded some respect. Perhaps you can make the same argument for the locality bias, but I guess that Rationalists (generally) either believe that you can't, or at least have a moral duty to optimize for the largest scope your computational power allows.

yeah, my model of the "us before them" question is that it is almost always globally optimal to cooperate, once a certain level of economic productivity is present. The safety that people are worried about is guaranteed not by maximizing their wealth but by minimizing their chances of death/starvation/conquest. Up to a point this means being strong and subjugating your neighbor (cf most of antiquity?) but eventually it means collaborating with them and including them in your "tribe" and extending your protection to them. I have no respect for anyone who argues to undo this, which is I think basically the ethos of the trump movement: by convincing everyone that they are under threat, they get people to turn on those that are actually working in concert with them (in order to enrich/empower themselves). It is a schema problem: we are so very very far away from an us vs. them world that it requires delusions to believe.

(...that said, progressivism has largely failed in dispelling this delusion. It is far too easy to feel as though progressivism/liberalism exists to prop up power hierarchies and economic disparities because in many ways it does, or has been co-opted to do that. I think on net it does not, but it should be much more cut-and-dry than it is. For that to be the case progressivism would need to find a way to effectively turn on its overlords/parasites, that is, rent-extracting capitalism and status-extracting moral elitism).

re: the first part of your reply. I sorta agree but I do think people do more extrapolation than you're saying on their own. The extrapolation is largely based on pattern-matching to known things: we have a rich literature (in the news, in art, in personal experience and storytelling) of failure modes of societies, which includes all kinds of examples of people inventing new moral rationalizations for things and using them to disregard personal morality. I think when people are extrapolating rationalists' ideas to find things that creep them out, they're largely pattern-matching to arguments they've seen in other places. But those arguments are, well, real arguments that require addressing.

And yeah, there are plenty of examples of people being afraid of things that today we think they should not have been afraid of. I tend to think that that's just how things go: it is the arc of social progress to figure out how to change things from unknown+frightening to known+benign. I won't fault anyone for being afraid of something they don't understand, but I will fault them for not being open-minded about it or being unempathetic or being cruel or not giving people chances to prove themselves.

All of this is rendered much more opaque and confusing by the fact that everyone places way too much stock in words, though. (e.g. the OP I was replying to who was taking all these criticisms of the rationalists at face-value). IMO this is a major trend that fucks royally with our ability as a society to make moral progress: we have come to believe that words supplant emotional intuition in a way that wrecks out ability to actually understand what people are upset about (I like to blame this trend for much of the modern political polarization). A small example of this is a case that I think everyone has experienced, which is a person discounting their own sense of creepiness from somebody else because they can't come up with a good reason to explain it and it feels unfair to treat someone coldly on a hunch. That should never have been possible: everyone should be trusting their hunches.

(which may seem to conflict with my preceding paragraph... should you trust your hunches or give people the chance to prove themselves? well, it's complicated, but it also really depends on what the result is. Avoiding someone personally because they creep you out is fine, but banning their way of life when it doesn't affect you at all is certainly not.)

One mistake you making is thinking that rationalists care more about people far away than people in their community. The reality is that they set the value of life the same for all.

If children around you are doing of an easily preventable disease, then yes, help them first! If they just need more arts programs, then you help the children dying in another country first.

That's not a mistake I'm making. Assuming you're talking about bog-standard effective altruists---by (claiming to) value the suffering of people far away as the same as those nearby, they're discounting the people around them heavily compared to other people. Compare to anyone else who values their friends and family and community far more than those far away. Perhaps they're not discounting them to less-than-parity---just less than they are for most people.

But anyway this whole model follows from a basic set of beliefs about quantifying suffering and about what one's ethical responsibilities are, and it answers those in ways most people would find very bizarre by turning them into a math problem that assigns no special responsibility to the people around you. I think that is much more contentious and gross to most people than EA thinks it is. It can be hard to say exactly why in words, but that doesn't make it less true.

To me, the non-local focus of EA/rationalism is, at least partially, a consequence of their historically unusual epistemology.

In college, I became a scale-dependent realist, which is to say, that I'm most confident of theories / knowledge in the 1-meter, 1-day, 1 m/s scales and increasingly skeptical of our understanding of things that are bigger/smaller, have longer/short timeframes, or faster velocities. Maybe there is a technical name for my position? But, it is mostly a skepticism about nearly unlimited extrapolation using brains that evolved under selection for reproduction at a certain scale. My position is not that we can't compute at different scales, but that we can't understand at other scales.

In practice, the rationalists appear to invert their confidence, with more confidence in quarks and light-years than daily experience.

> no special responsibility to the people around you

Musing on the different failure-directions: Pretty much any terrible present thing against people can be rationalized by arguing that one gadzillion distant/future people are more important. That includes religious versions, where the stakes of the holy war may presented as all of future humanity being doomed to infinite torment. There are even some cults that pitch it retroactively: Offer to the priesthood to save all your ancestors who are in hell because of original sin.

The opposite would be to prioritize the near and immediate, culminating in a despotic god-king. This is somewhat more-familiar, we may have more cultural experience and moral tools for detection and prevention.

A check on either process would be that the denigrated real/nearby humans revolt. :p

> they're discounting the people around them heavily compared to other people

This statement of yours makes no sense.

EAs by definition are attempting to remove the innate bias that discounts people far away by instead saying all lives are of equal worth.

>turning them into a math problem that assigns no special responsibility to the people around you

All lives are equal isn't a math problem. "Fuck it blow up the foreigners to keep oil prices low" is a math problem, it is a calculus that the US government has spent decades performing. (One that assigns zero value to lives outside the US.)

If $100 can save 1 life 10 blocks away from me or 5 lives in the next town over, what kind as asshole chooses to let 5 people die vs 1?

And since air travel is a thing, what the hell does "close to us" mean?

For that matter, from a purely selfish POV, helping lift other nations up to become fully advanced economies is hugely beneficial to me, and everyone on earth, in the long run. I'm damn thankful for all the aid my country gave to South Korea, the number of scientific advances that have come out of SK damn well paid for any tax dollars my grandparents paid on many orders of magnitude times over.

> It can be hard to say exactly why in words, but that doesn't make it less true.

This is the part where I shout racism.

Because history has shown it isn't about people being far or close in distance, but rather in how those people look.

Americans have shot down multiple social benefit programs because, and these are what people who voted against those programs directly said was their reasons "white people don't want black people getting the same help white people get."

Whites in America have voted, repeatedly, to keep themselves poor rather than lift themselves and black families out of poverty at the same time.

Of course Americans think helping people in Africa is "weird".

> If $100 can save 1 life 10 blocks away from me or 5 lives in the next town over, what kind as asshole chooses to let 5 people die vs 1?

The thing about strict-utilitarian-morality is that it can't comprehend any other kind of morality, because it evaluates the morality of... moralities... on its own utilitarian basis. And then of course it wins over the others: it's evaluating them using itself!

There are entirely different ethical systems that are not utilitarian which (it seems) most people hold and innately use (the "personal morality" I'm talking about in my earlier post). They are hard to comprehend rationally, but that doesn't make them less real. Strict-utilitarianism seems "correct" in a way that personal morality does not because you are working from a premise "only things that I can understand like math problems can be true". But what I observe in the world is that people's fear of the rationalist/EA mindset comes from the fact that they empirically find this way of thinking to be insidious. Their morality specifically disagrees with that way of thinking: it is not the case that truth comes from scrutable math problems; that is not the point of moral action to them.

The EA philosophy may be put as "well sure but you could change to the math-problem version, it's better". But what I observe is that people largely don't want to. There is a purpose to their choice of moral framework; it's not that they're looking at them all in a vacuum and picking the most mathematically sound one. They have an intrinsic need to keep the people around them safe and they're picking the one that does that best. EA on the other hand is great if everyone around you is safe and you have lots of extra spending money and what you're maximizing for is the feeling of being a good person. But it is not the only way to conceive of moral action, and if you think it is, you're too inside of it to see out.

I'll reiterate I am trying to describe what I see happening when people resist and protest rationalism (and why their complaints "miss" slightly---because IMO they don't have the language to talk about this stuff but they are still afraid of it). I'm sympathetic to EA largely, but I think it misses important things that are crippling it, of the variety above: an inability to recognize other people's moralities and needs and fears doesn't make them go away; it just makes them hate you.

> The thing about strict-utilitarian-morality is that it can't comprehend any other kind of morality, because it evaluates the morality of... moralities... on its own utilitarian basis.

I can comprehend them just fine, but I have a deep seated objection to any system of morality that leaves behind giant piles of dead bodies. We should be trying to minimize the size of the pile of dead bodies (and ideally eliminate the pile altogether!)

Any system or morality that boils down to "I don't care about that pile of dead bodies being huge because those people look different" is in fact not a system morality at all.

Well, you won't find anyone who disagrees with you here. No such morality is being discussed.

The job of a system of morality is to synthesize all the things we want to happen / want to prevent happening into a way of making decisions. One such thing is piles of dead bodies. Another is one's natural moral instincts, like their need to take care of their family, or the feeling of responsibility to invest time and energy into improving their future or their community or repairing justice or helping people who need help, or to attend to their needs for art and meaning and fun and love and respect. A coherent moral system synthesizes these all and figures out how much priority to allocate to each thing in a way that is reasonable and productive.

Any system of morality that takes one of these criteria and discards the rest of them is not a system of morality at all, in the very literal sense that nobody will do it. Most people won't sell out one of their moral impulses for the others, and EA/rationalism feels like it asks them too, since it asks them to place zero value in a lot of things that they inherently place moral value in, and so they find it creepy and weird. (It doesn't ask that explicitly; it asks it by omission. By never considering any other morality and being incapable of considering them, because they are not easily quantifiable/made logical, it asks you to accept a framework that sets you up to ignore most of your needs.)

My angle here is that I'm trying to describe what I believe is already happening. I'm not advocating it; it's already there, like a law of physics.

Perhaps part of it is that local action can often be an order of magnitude more impactful than the “equivalent” action at a distance. If you volunteer in your local community, you not only have fine-grained control over the benefit you bestow, you also know for a fact that you’re doing good. Giving to a charity that addresses an issue on the other side of the world doesn’t afford this level of control, nor this level of certainty. For all you know most of the donation is being embezzled.

I think another part of it is a sort of healthy nativism or in-group preference or whatever you want to call it. It rubs people the wrong way when you say that you care about someone in a different country as much as you care about your neighbors. That’s just…antisocial. Taken to its logical conclusion, a “rationalist” should not only donate all of their disposable income to global charities, they should also find a way to steal as much as possible from their neighbors and donate that, too. After all, those. Holden in Africa need the money much more than their pampered western neighbors.

> Humans are generally better at perceiving threats than they are at putting those threats into words.

Not only that, but this is exactly the kind of scenario where we should be giving those signals the most weight: The individual estimating whether to join up with a tribe. (As opposed to, say, bad feelings about doing calculus.)

Not only does it involve humans-predicting-humans (where we have a rather privileged set of tools) but there have been millions of years of selective pressure to be decent at it.

Oh this is a great breakdown, I totally agree. I feel there’s a morality hierarchy that’s akin to Maslow’s hierarchy of needs.

People benefit from a sense of a family, a sense of community. It helps us feel more secure, both personally and to our loved ones.

I think the more I view things through this lenses the downstream benefits I see.

This is a remarkably well started comment.

One thing I'd like to add though is that I do think there is an additional piece being discarded irrationally. They tend to highly undervalue everything you're describing. Humans aren't Vulcans. By being so obsessed with the risks of paperclip-maximizing-robots they devalue the risks of humans being the irrational animals they are.

This is why many on the left criticize them for being right wing. Not because they are, well some might be, but because they are incredibly easy to distract from what is being communicated by focusing too much on what is being said. That might be a bad phrasing but what I mean is that when you look at this piece from last year about prison sentence length and crime rates by Scott Alexander[0] nothing he says is genuinely unreasonable. He's generally evaluating the data fairly and rationally. Some might disagree there but that's not my point. My point is that he's talking to a nonexistant group. The right largely believes that punishment is the point of prison. They might _say_ the goal is to reduce crime, but they are communicating based on a set of beliefs that strongly favors punitive measures for their own sake. This causes a piece like that to miss the forest through the trees and can be seen by those on the left as functionally right wing propaganda.

Most people are not rational. Maybe some day they will be but until then it is dangerous to assume and act as if they are. This makes me see the rationalists as actually rather irrational.

0: https://www.astralcodexten.com/p/prison-and-crime-much-more-...

This is a fantastic comment, but it also underscores the thing I really dislike about most internet conversations about philosophy: a lack of reading prior art. Utilitarian morality certainly isn't a new concept though the rationalists today are some of its strongest standard bearers. But around the time when utilitarianism was starting to take hold, rigorous philosophical thinking had exactly the debates that come up here.

Part of the reason I enjoy rationalist discourse more is because, even if they are unabashedly utilitarian, they try to rigorously derive philosophy. Most internet discourse on philosophy is, as you say, just vaguely derived around gut feelings. But philosophy can and has been thought of rigorously. Virtue ethics and continental morality are both schools of thought that reject utilitarian ethics but are much more meaty than the sort of internet "no but my neighbors" intuition that you see in full force, and the weird insistence that these internet commenters continue to use their vague moral intuition without being rigorous about their own thoughts.

Found the person who doesn't have this skill issue! :) Very well put.

I never quite realized the connection between this creepy feeling and the Bleak House criticism of far-flung philanthropy, until this comment.

heh. It would not be an overstatement to say I've been working on better putting the issue with the rationalists into words for ten years. glad it's resonating with some people.

Ever had a conversation with someone who literally cannot reexamine their base principles? Like, it's functionally impossible for them? That's everyone's central criticism of rationalists.

Being able to do that is pretty much "entry level cognition" for a lot of us. You should be doing that yourself and doing it all the time if you want to play with the big kids.

One of the things I really miss about the old nerds-only programmer's pit setup was the amount of room we had for instruction, especially regarding social issues. The scenes from the college department in Wargames were really on the nose, but highlight a form of education that was unavoidable if you couldn't just dip out of a conversation.

What a deeply confused criticism of rationalists, who are possibly the only meaningful social group in existence to celebrate changing their minds in response to new evidence.

It's truly fascinating how this thread is full of demonstrations of the exact phenomenon being discussed!

Accused of being unable to reexamine your base principles, you respond:

> rationalists, who are possibly the only meaningful social group in existence to celebrate changing their minds in response to new evidence

Which is exactly the kind of base principle that could use some reexamination.

That's an empirical claim, not a "base principle" (whatever that is). And, uh, yeah, I agree that I am not re-examining my beliefs in response to a bunch of random unsupported accusations by people who are demonstrably not familiar with the thing they're attacking. You should update your beliefs in response to evidence, not in response to social attacks.

Again, this is exactly what we're talking about. You expressed a belief about "rationalists" and the history of people changing their minds about things, I suggested maybe you should reexamine that belief, and you said nope absolutely not. (In a condescending way.) This is the kind of engagement that is being critiqued in this thread.

Read other comments then. Not everyone is saying “they think they are better than us”.

The primary issues as others have noted is they focus on people going to the highest paying jobs without much care for morality of the jobs. Ergo they are fine being net negatives in terms of their work and philosophy.

All they do is donate money. Donations don’t fix society. Nothing changes structurally. No root problems are looked at.

They largely ignore capitalism’s faults or when I’ve seen them talk about, it’s done in a way of superficially decrying capitalist issues but then largely going along with them. Which ties into how they focus on high paying jobs regardless of morality (I’m exaggerating here but the overall point is correct).

—

HN is not intelligent when it comes to politics or the world. The avg person here is a western chauvinist with little political knowledge but a defensive ego about it. No need to be sad about this comment page.

Well put.

Thanks

Or maybe you are demonstrating the exact failure to demonstrate introspective humility that people are talking about?

Having just moved to the Bay Area, I've met a few AI "safety researchers" who seems to come from this EA / Rationalist camp, and they all behave more like preachers than thinkers / academics / researchers.

I don't think any "Rationalists" I ever met would actually consider concepts like scientific method...

> I don't think any "Rationalists" I ever met would actually consider concepts like scientific method...

In that case I don't think you've met any of the people under discussion.

maybe, but I Stanford scientists are a different breed of rationalists, and not Rationalists

> They remind me of the "Effective Altruism" crowd who get completely wound up in these hypothetical logical thought exercises and end up coming to insane conclusions that they feel trapped in because they got there using pure logic. Not realizing that their initial conditions were highly artificial so any conclusion they reach is only of academic value.

Do you have examples of that? I have a different perception, most of the EAs I've met are very grounded and sharp.

For example the most recent issue of their newsletter: https://us8.campaign-archive.com/?e=7023019c13&u=52b028e7f79...

I'm not sure where there are any “hypothetical logical thought exercises” that “end up coming to insane conclusions” in there.

For the first part where you say “not realizing that their initial conditions were highly artificial so any conclusion they reach is only of academic value” this is quite the opposite of my experience with them. They are very receptive to criticism and reconsider their point of view in reaction to that.

They are generally well-aware of the limits of data-driven initiatives and the dangers of indulging into purely abstract thinking that can lead to conclusions that indeed don't make sense.

The confluence of Bay Area rationalism and academic philosophy means a lot of other EA space is given to discussing hypotheticals in longwinded forum posts, blogs and papers. Some of those are well-trod utilitarian debates, others take it towards uniquely EA arguments like asserting that given that there could be as many as 10^31 future humans, essentially anything which claims to reduce existential risk - no matter how implausible the mechanism - has higher expected value than doing stuff that would certainly save human lives. An apparently completely unironic forum argument asked their fellow EAs to consider the possibility that given various heroic assumptions, the sum total of the suffering caused to mosquitos by anti-malaria nets might in fact be larger than the suffering caused by malaria they prevent.

The newsletter is of course far more to the point than that, but even then you'll notice half of it is devoted to understanding the emotional state and intentions of LLMs...

It is of course entirely possible to identify as an "Effective Altruist" whilst making larger-than-averaged donations to charities with rigorous efficacy metrics and otherwise being completely normal, but that's not the centre of EA debate....

> that's not the centre of EA debate or culture....

EAs gave $1,886,513,058 through GiveWell[1], and there is 0 AI stuff in there (you can search in the linked Airtable spreadsheet).

There is also a whole movement for doing a lifetime commitment to give 10% of your earnings to charity. 9,880 people took the pledge so far[2].

[1] https://airtable.com/appGuFtOIb1eodoBu/shr1EzngorAlEzziP/tbl...

[2] https://www.givingwhatwecan.org/pledge

Sure, but I'd say that the philosophical musings of EA's leadership, events, university outreach centres, and official forum were more representative of "the centre of EA debate or culture" than pledge signatories, even though I've got wayyy more time for stuff like the Giving Pledge.

GiveWell continues to plow its own rigorous international development-focused furrow, but it's cofounder, once noted for calling out everyone else for lack of rigour in their evidence base, has moved on to fluffy essays about how this is probably "the most important century" because it's either AI armageddon or maybe his wife's $61B startup will save us all...

As Adam Becker shows in his book, EAs started out being reasonable "give to charity as much as you can, and research which charities do the most good" but have gotten into absurdities like "it is more important to fund rockets than help starving people or prevent malaria because maybe an asteroid will hit the Earth, killing everyone, starving or not".

It's also not a very big leap from "My purpose is to do whatever is the greatest good" to "It doesn't matter if I hurt people as long as the overall net result is good (by some arbitrary standard)"

I think this is the key comment so far.

99% of effective altruists and rationalists agree that you shouldn't hurt people as part of some complicated scheme to do good. For example, here is eliezer yudkowsky in 2008 saying exactly that, and explaining why it's true: https://www.lesswrong.com/posts/K9ZaZXDnL3SEmYZqB/ends-don-t...

I believe they believe that, on its face.

I also believe that idealistic people will go to great lengths to convince themselves that their desired outcome is, in fact, the moral one. It starts by saying things like, "Well, what is harm, actually..." and then constructing a definition that supports the conclusions they've already arrived at.

I'm quite sure Sam Bankman-Fried did not believe he was harming anybody when he lost/stole/defrauded his investors and depositors' money.

Like an very old dude once said: "No one is willfully evil."

This isn’t a hypothetical leap either. This thinking directly lead to the murders committed by the zizians.

How do they escape the reality that the Earth will one day be destroyed, and that it's almost certainly impossible to ever colonize another planetary system? Just suicide out?

If you value maximizing the number of human lives that are lived, then even “almost certainly impossible” is enough to justify focusing a huge amount of effort on that. Maybe interstellar colonization is a one in a million shot, but it would multiply the number of human lives by billions or trillions or more.

Is the argument that we should try to do things that will benefit our theoretical and theoretically multitudinous descendants? Or is it that just taking action to make their existence more likely is a moral good? Because the latter is just brain dead.

Good question. I think it has to be the latter, given the immense time involved. You can make a connection between driving progress in certain areas today and increasing the odds that humanity eventually colonizes the stars. I don’t think you can make any connection with how well off those far-future humans will be.

If that's what's meant, it's a hilarious perversion of utilitarianism.

It seems very odd to criticize the group that most reliably and effectively funds global health and malaria prevention for this.

What is your alternative? What's your framework that makes you contribute to malaria prevention more or more effectively than EAs do? Or is the claim instead that people should shut down conversation within EA that strays from the EA mode?

The simple answer is you don't need a "framework" -- plain empathy for the less fortunate is good enough. But if the EA's actually want to do something about malaria (although the Gates Foundation does much, much more in that regard than the Centre for Effective Altruism), more power to them. But as Becker notes from his visits to the Centre, things like malaria and malnutrition are not the primary focus of the centre.

I wish it were, but it's clearly not enough. There are plenty of people with healthy emotional empathy in the world, and yet children still die of easily preventable diseases.

I am plenty happy to simp for the Gates foundation, but I think it's important to acknowledge that becoming Bill Gates to support charity is not a strategy the average person can replicate. The question for me is how do I live my life to support the causes I care about, not who lives more impressive lives than me.

EA people gave a total of $817,276,989 to malaria initiatives through GiveWell[1][2].

How much more do they need to give before you will change your mind about whether “EA's actually want to do something about malaria”?

[1] https://www.givewell.org/all-grants-fund

[2] https://airtable.com/appGuFtOIb1eodoBu/shr1EzngorAlEzziP/tbl...

I've used GiveWell for donations and don't consider myself an Effective Altruist. Does GiveWell get to count for the just the EA community?

By analogy, if a Catholic Church created a charity for curing malaria, and I donated money to it, that wouldn't make me Catholic. But still the existence of the charity, especially if people donated over a billion dollars to it, would be a credible argument against people saying "Catholics do nothing about curing malaria". Does that make sense?

I think the group that most reliably and effectively funds global health -- at least in terms of total $ -- would be the United Nations, or perhaps the Catholic Church, or otherwise one national government or another.

If you exclude "nations" then it does look to be the Church: "The Church operates more than 140,000 schools, 10,000 orphanages, 5,000 hospitals and some 16,000 other health clinics". Caritas, the relevant charitable umbrella organization, gives $2-4b per year on its own, and that's not including the many, many operations run by religious orders not under that umbrella, or by the hundreds of thousands of parishes around the world (most of which operate charitable operations of their own).

And yet, rationalists are totally happy criticizing the Catholic Church -- not that I'm complaining, but it seems a bit hypocritical.

Rationalists and EAs spend far more time praising the Catholic Church and other religious groups than criticizing them - since they spend essentially no time criticizing them, and do occasionally praise them.

I appreciate the good these organizations do, but I don't think that's the right measure of it. A person wouldn't in expectation serve global health better by becoming Catholic than by joining EA. That Catholicism is large isn't the same as them being effective at solving malaria. EA is tiny relative to the Church but still manages to support within an order of magnitude the funding you mentioned here, with exact numbers depending on how you count.

Similarly, it's not like government funding is an overlooked part of EA. Working on government and government aid programs is something EA talks about, high leverage areas like policy especially. If there's a more standard government role that an individual can take that has better outcomes than what EAs do, that would be an important update and I'd be interested in hearing it. But the criticism that EA is just not large enough is hard to action on, and more of a work in progress than a moral failing.

There's a lot of weird ideas on the fringes of EA...but they are on the fringes. Everyone is eye rolling at shrimp guy.

https://www.greaterwrong.com/posts/5JB4dn7hEvZCYDvCe/a-very-...

https://www.greaterwrong.com/posts/MHBJ436AiKyN83DCB/thinkin...

https://www.greaterwrong.com/posts/58ajesm2C38wi3WSJ/insect-...

One example is Newcomb's problem. It presupposes a ridiculous scenario where a godlike being acts irrationally and then people try to base their life around "winning" the game that will never ever happen to them.

> their initial conditions were highly artificial

There has to be (or ought to be) a name for this kind of epistemological fallacy, where in pursuit of truth, the pursuit of logical sophistication and soundness between starting assumptions (or first principles) and conclusions becomes functionally way more important than carefully evaluating and thoughtfully choosing the right starting assumptions (and being willing to change them when they are found to be inconsistent with sound observation and interpretation).

Yes, there's a name for it. They're dumbasses.

“[...] Clevinger was one of those people with lots of intelligence and no brains, and everyone knew it except those who soon found it out. In short, he was a dope." - Joseph Heller, Catch-22 https://www.goodreads.com/quotes/7522733-in-short-clevinger-...

You mean people projecting this problem onto them with zero evidence beyond their own preconceptions and an occasional stereotype they read online?

HN should be better than this.

Let's read the thread: "There has to be (or ought to be) a name for this kind of epistemological fallacy, where in pursuit of truth, the pursuit of logical sophistication and soundness between starting assumptions (or first principles) and conclusions becomes functionally way more important than carefully evaluating and thoughtfully choosing the right starting assumptions (and being willing to change them when they are found to be inconsistent with sound observation and interpretation)."

Can people suffer from that impairment? Is that possible? If not, please explain how wrong assumptions can be eliminated without actively looking for them. If the impairment is real, what would you call its victims? Pick your own terminology.

Name calling may be helpful in some contexts, but I doubt it's persuasive to the people in question.

I think a huge part of the subtext to this conversation is that nothing is persuasive to the people in question. How many people in this thread have come down off the mountain to explain a point of view that's just bouncing off some dude who's never needed to develop a feedback loop for his thinking?

Calling someone a dumbass in this situation is a kindness, because the assumption is that they're capable of not being one with a little self-reflection.

> Calling someone a dumbass in this situation is a kindness

First they laugh...

Kindness means you're seeking the good of another -- it's noticing the needs of others and prioritizing their well-being.

I can't see how insulting someone pursues their well-being in any way.

That may work for literature, but I was hoping for something more precise.

Bullet biting.

Maybe "appeal to authority" covers it. The "rationality" or "intellect" or even "brilliance" of some claim's originator are fun things to talk about. But (even taking the claims at face value), those things aren't logically connected to whether the claim is actually true.

Or "garbage in, garbage out"? But that's really about computers. And maybe it is name calling. Is there no standard name for this? Or not in English?

Maybe it's similar to the "Good Student" picture. Bright within a given assignment, but taking the assignment to be immutable, or taking no interest in where the assignment comes from.

Or I've heard a saying "nobody will pay you to solve problems that they've already defined clearly."

They _are_ the effective altruism crowd.

There is a great recent book by Adam Becker "More Everything Forever" that deals with the overlapping circles of "effective altruists", "rationalists", and "accelerationists". He's not very sympathetic to the movements as he sees them as mostly rationalizing what their adherents wanted to do anyway -- funding things like rockets and AI over feeding the needy because they see the former as helping more people in the future than dealing with real problems today.

People who confuse the map for the territory.

The notion that our moral obligation somehow demands we reduce the suffering of wild animals in an ecosystem, living their lives as they have done since predation evolved and as they will do long after humans have ceased to be, is such a wild misunderstanding of who we are and what we are and what the universe is. I love my Bay Area friends. To quote the great Gwen Stefani, “This sh!t is bananas.”

> They remind me of the "Effective Altruism" crowd

Isn't there a lot of overlap between the two groups?

I recently read a great book that examines these various groups and their commonality: More Everything Forever: AI Overlords, Space Empires, and Silicon Valley's Crusade to Control the Fate of Humanity by Adam Becker. Highly recommended.

Except with Effective Altruism (EA), its not pure logic.

Logic requires properties of metaphysical objectivity.

If you use the true meaning of words it would be called irrationality, delusion, sophism, or fallacy when such things are claimed true when in fact they are false.

Incidentally a good book on logic is the best antidote to that type of thinking. Once you learn the difference between a valid and a sound argument and then realize just how ambiguous every English sentence is the idea that just because you have a logical argument you have something useful in everyday life becomes laughable rather quickly.

I also think the ambiguity of meaning in natural language is why statistical llms are so popular with this crowd. You don't need to think about meaning and parsing. Whatever the llm assumes is the meaning is whatever the meaning is.

> Once you learn the difference between a valid and a sound argument and then realize just how ambiguous every English sentence is the idea that just because you have a logical argument you have something useful in everyday life becomes laughable rather quickly.

Ironically, that reminds me of "37 Ways That Words Can Be Wrong", written by Eliezer Yudkowsky in 2008...

>They remind me of the "Effective Altruism" crowd who get completely wound up in these hypothetical logical thought exercises and end up coming to insane conclusions that they feel trapped in because they got there using pure logic

I have read Effective Altruists like that. But I also remember seeing a lot of money donated to a bunch of really decent sounding causes because someone spent 5 minutes asking themselves what they wanted their donation to maximise, decided on "Lives saved" and figured out who is doing the best at that.

lol, I love that you identified the exact correct academic term for this phenomenon.

I really appreciate this comment. Now in my 30s, I was not even aware of EA or Rationalist groups in my 20s but your description of a person high on the smell of their own farts and putting logic as king was me. The world is so chaotic that it felt good and right that I could lean on something "true" like logic. It took me to the "correct" answers according to logic but very misguided answers as I see them now. Humans are driven by emotions and the world is probabilistic so relying so heavily on logic just seems so misguided now that I'm older.

By the way, "the world is probabilistic" is one of the key lessons of the rationality community. (Infinite certainty would require infinite amount of evidence, according to the Bayes' Theorem.) When you find someone on internet trying to assign probabilities to their beliefs, chances are you have met a rationalist.

I don’t have anything to add to the discussion except that EA was the very first thing that popped into my mind when I read Rationalists.

I'd recommend listening to the behind the bastards episode on some of these folks, it is wild https://podcasts.apple.com/us/podcast/part-one-the-zizians-h...

This is a perfect example of a Hacker News hot-take, you complain about their attitude while making zero reference to their actual accomplishments or failures. No one will downvote you for posting something factually incorrect, because you didn't post any facts at all.

For anyone else reading, a good example of what EAs do can be seen with the GiveWell charity: https://www.givewell.org/top-charities-fund

Lots of anti-malaria and vitamin stuff (as a cheap way to save lots of lives). There are also tons of EA animal charities too, such as Humane League: https://thehumaneleague.org/our-impact

A weird cult doing helpful community service doesn't cancel out the harm of them being a cult.

If they want to donate to charity, they can just donate. You don't gotta make a religion out of it.

Just like the other guy, you skip the part where you actually name a harm done by them, because then people could easily shoot down your comment. Or name how EAs are a religion, I haven't even seen that troll before.

> They remind me of the "Effective Altruism" crowd

Honestly thought they were the same people

Not all rationalists are EAs, and not all EAs are rationalists, but there are many people who are both. The very idea of using evidence and numbers to choose the optimal charity has a rationalist vibe. But of course, one can donate to an effective charity without being interested in LessWrong-style rationality.

https://www.lesswrong.com/ and https://forum.effectivealtruism.org/ run on the same software, and some people have an account at both of them, but they are generally separate web communities.

Absolutely agree. The whole movement is unbearably naive