I get it, I enjoyed being told I'm a super genius always right quantum physicist mathematician by the girls at Stanford too. But holy hell man, have some class, maybe consider there's more good to be done in rural Indiana getting some dirt under those nails..

but I don't know enough about it, I'm just trolling.

I find it sadly hilarious to watch academic types fight over meaningless scraps of recognition like toddlers wrestling for a toy.

That said, I enjoy some of the rationalist blog content and find it thoughtful, up to the point where they bravely allow their chain of reasoning to justify antisocial ideas.

In real life, the conversation too often ends up being, "This has to be wrong, and you're an obnoxious nerd for bothering me with it," versus, "You don't understand my argument, so I am smarter, and my conclusions are brilliantly subversive."

It frequently reduces complex problems into comfortable oversimplifications.

Maybe you don't think that is real wisdom, and maybe that's sort of your point, but then what does real wisdom look like? Should wisdom make you considerate of the multiple contexts it does and doesn't affect? Maybe the issue is we need to better understand how to evaluate and use wisdom. People who truly understand a piece of wisdom should communicate deeply rather than parroting platitudes.

Also to be frank, wisdom is a way of controlling how others perceive a problem, and is a great way to manipulate others by propping up ultimatums or forcing scope. Much of past wisdom is unhelpful or highly irrelevant to modern life.

e.g. "Good things come to those who wait."

Passive waiting rarely produces results. Initiative, timing, and strategic action tend to matter more than patience.

Both our biology and other complex human affairs like societies and cultures evolved organically over long periods of time, responding to their environments and their competitors, building bit by bit, sometimes with an explicit goal but often without one.

One can learn a lot from unicellular organisms, but won’t probably be able to reason from them all the way to an elephant. At best, if we are lucky, we can reason back from the elephant.

Actually, neither do Rationalists, but instead they cosplay at being rational.

What do you mean? The biologists I've had the privilege of working with absolutely do try to. Obviously some work at a higher level of abstraction than others, but I've not met any who apply any magical thinking to the actual biological investigation. In particular (at least in my milieu), I have found that the typical biologist is more likely to consider quantum effects than the typical physicist. On the other hand (again, from my limited experience), biologists do tend to have some magical thinking about how statistics (and particularly hypothesis testing) works, but no one is perfect.

Reasoning from first principles cannot span very far in reality, as for starters the complexity of the argument quickly overwhelms our capacity for it. Its numerous other limits have been well-documented.

Logicomix, Gödel Escher Bach are some common entry points.

I'm kinda new here but am surprised I haven't seen this book mentioned more. Maybe I just haven't seen it or it's old news but it seems right up HNs alley.

Biologists stand out because they have already given up on that idea. They may still seek to simplify complex things by refining principles of some kind, but it's a "whatever stories work best" approach. More Feyerabend, less Popper. Instead of axioms they have these patterns that one notices after failing to find axioms for a while.

The largest of the finite simple groups (themselves objects of study as a means of classifying other, finite but non-simple groups, which can always be broken down into simple groups) is the Monster Group -- it has order 808017424794512875886459904961710757005754368000000000, and cannot be reduced to simpler "factors". It has a whole bunch of very interesting properties which thus can only be understood by analyzing the whole object in itself.

Now whether this applies to biology, I doubt, but it's good to know that limits do exist, even if we don't know exactly where they'll show up in practice.

Examples that come to mind: statistical modelling (reduction to nonparametric models), protein folding (reduction to quantum chemistry), climate/weather prediction (reduction to fluid physics), human language translation (reduction to neural networks).

Reductionism is not that useful as a theory building tool, but reductionist approaches have a lot of practical value.

I am not sure in what sense folding simulations are reducable to quantum chemistry. There are interesting 'hybrid' approaches where some (limited) quantum calculations are done for a small part of the structure - usually the active site I suppose - and the rest is done using more standard molecular mechanics/molecular dynamics approaches.

Perhaps things have progressed a lot since I worked in protein bioinformatics. As far as I know, even extremely short simulations at the quantum level were not possible for systems with more than a few atoms.

If you're looking for insults, and declaring the whole conversation a "culture war" as soon as you think you found one, (a) you'll avoid plenty of assholes, but (b) in the end you will read whatever you want to read, not what the thoughtful people are actually writing.

What you are mentioning is called western reductionism by some.

In the western world it does map to Plato etc, but it is also a problem if you believe everything is reducible.

Under the assumption that all models are wrong, but some are useful, it helps you find useful models.

If you consider Laplacian determinism as a proxy for reductionism, Cantor diagonalization and the standard model of QM are counterexamples.

Russell's paradox is another lens into the limits of Plato, which the PEM assumption is based on.

Those common a priori assumptions have value, but are assumptions which may not hold for any particular problem.

Reduce a computer's behavior to its hardware design, state of RAM, and physical laws. All those voltages make no sense until you come up with the idea of stored instructions, division of the bits into some kind of memory space, etc. You may say, you can predict the future of the RAM. And that's true. But if you can't read the messages the computer prints out, then you're still doing circuits, not software.

Is that reductionist approach providing valuable insight? YES! Is it the whole picture? No.

This warning isn't new, and it's very mainstream. https://www.tkm.kit.edu/downloads/TKM1_2011_more_is_differen...

This is true for science and rationalism itself. Part of the problem is that "being rational" is a social fashion or fad. Science is immensely useful because it produces real results, but we don't really do it for a rational reason - we do it for reasons of cultural and social pressures.

We would get further with rationalism if we remembered or maybe admitted that we do it for reasons that make sense only in a complex social world.

Thanks, I might actually go do this :) I recently got exposed to a very persuasive form of "rationalism is a social construct" by reading "Alchemy" by Rory Sutherland. But a theme in these comments is that a lot of these ideas are just recycled from philosophers and that the philosophers were less likely to try and induct you into a cult.

Both advocate the principle of putting out a bold conjecture, test, learn, adapt.

I originally came to this critique via Heidegger, who argues that enlightenment thinking essentially forgets / obscures Being itself, a specific mode of which you experience at this very moment as you read this comment, which is really the basis of everything that we know, including science, technology, and rationality. It seems important to recover and deepen this understanding if we are to have any hope of managing science and technology in a way that is actually beneficial to humans.

Maybe, but generally speaking, if I think people are playing around with technology which a lot of smart people think might end humanity as we know it, I would want them to stop until we are really sure it won't. Like, "less than a one in a million chance" sure.

Those are big stakes. I would have opposed the Manhattan Project on the same principle had I been born 100 years earlier, when people were worried the bomb might ignite the world's atmosphere. I oppose a lot of gain-of-function virus research today too.

That's not a point you have to be a rationalist to defend. I don't consider myself one, and I wasn't convinced by them of this - I was convinced by Nick Bostrom's book Superintelligence, which lays out his case with most of the assumptions he brings to the table laid bare. Way more in the style of Euclid or Hobbes than ... whatever that is.

Above all I suspect that the Internet rationalists are basically a 30 year long campaign of "any publicity is good publicity" when it comes to existential risk from superintelligence, and for what it's worth, it seems to have worked. I don't hear people dismiss these risks very often as "You've just been reading too many science fiction novels" these days, which would have been the default response back in the 90s or 2000s.

I've recently stumbled across the theory that "it's gonna go away, just keep your head down" is the crisis response that has been taught to the generation that lived through the cold war, so that's how they act. That bit was in regards to climate change, but I can easily see it apply to AI as well (even though I personally believe that the whole "AI eat world" arc is only so popular due to marketing efforts of the corresponding industry)

I don't buy the marketing angle, because it doesn't actually make sense to me. Fear draws eyeballs, sure, but it just seems otherwise nakedly counterproductive, like a burger chain advertising itself on the brutality of its factory farms.

It’s rather more like the burger chain decrying the brutality as a reason for other burger chains to be heavily regulated (don’t worry about them; they’re the guys you can trust and/or they are practically already holding themselves to strict ethical standards) while talking about how delicious and juicy their meat patties are.

I agree about the general sentiment that the technology is dangerous, especially from a “oops, our agent stopped all of the power plants” angle. Just... the messaging from the big AI services is both that and marketing hype. It seems to get people to disregard real dangers as “marketing” and I think that’s because the actual marketing puts an outsized emphasis on the dangers. (Don’t hook your agent up to your power plant controls, please and thank you. But I somehow doubt that OpenAI and Anthropic will not be there, ready and willing, despite the dangers they are oh so aware of.)

I'm glad you ran with my burger chain metaphor, because it illustrates why I think it doesn't work for an AI company to intentionally try and advertise themselves with this kind of strategy, let alone ~all the big players in an industry. Any ordinary member of the burger-eating public would be turned off by such an advertisement. Many would quickly notice the unsaid thing; those not sharp enough to would probably just see the descriptions of torture and be less likely on the margin to go eat there instead of just, like, safe happy McDonald's. Analogously we have to ask ourselves why there seems to be no Andreessen-esque major AI lab that just says loud and proud, "Ignore those lunatics. Everything's going to be fine. Buy from us." That seems like it would be an excellent counterpositioning strategy in the 2025 ecosystem.

Moreover, if the marketing theory is to be believed, these kinds of psuedo-ads are not targeted at the lowest common denominator of society. Their target is people with sway over actual regulation. Such an audience is going to be much more discerning, for the same reason a machinist vets his CNC machine advertisements much more aggressively than, say, the TVs on display at Best Buy. The more skin you have in the game, the more sense it makes to stop and analyze.

Some would argue the AI companies know all this, and are gambling on the chance that they are able to get regulation through and get enshrined as some state-mandated AI monopoly. A well-owner does well in a desert, after all. I grant this is a possibility. I do not think the likelihood of success here is very high. It was higher back when OpenAI was the only game in town, and I had more sympathy for this theory back in 2020-2021, but each serious new entrant cuts this chance down multiplicatively across the board, and by now I don't think anyone could seriously pitch that to their investors as their exit strategy and expect a round of applause for their brilliance.

note, my assumption is not that the bomb would not have been developed. Only that by opposing the manhattan project the USA would not have developed it first.

Take this all with more than a few grains of salt. I am by no means an expert in this territory. But I don't shy away from thinking about something just because I start out sounding like an idiot. Also take into account this is post-hoc, and 1940 Manhattan Project me would obviously have had much, much less information to work with about how things actually panned out. My answer to this question should be seen as separate to the question of whether I think dodging the Manhattan Project would have been a good bet, so to speak.

Most historians agree that Japan was going to lose one way or another by that point in the war. Truman argued that dropping the bomb killed fewer people in Japan than continuing, which I agree with, but that's a relatively small factor in the calculation.

The much bigger factor is that the success of the Manhattan Project as an ultimate existence proof for the possibility of such weaponry almost certainly galvanized the Soviet Union to get on the path of building it themselves much more aggressively. A Cold War where one side takes substantially longer to get to nukes is mostly an obvious x-risk win. Counterfactual worlds can never be seen with certainty, but it wouldn't surprise me if the mere existence proof led the USSR to actually create their own atomic weapons a decade faster than they would have otherwise, by e.g. motivating Stalin to actually care about what all those eggheads were up to (much to the terror of said eggheads).

This is a bad argument to advance when we're arguing about e.g. the invention of calculus, which as you'll recall was coinvented in at least 2 places (Newton with fluxions, Liebniz with infinitesimals I think), but calculus was the kind of thing that could be invented by one smart guy in his home office. It's a much more believable one when the only actors who could have made it were huge state-sponsored laboratories in the US and the USSR.

If you buy that, that's 5 to 10 extra years the US would have had in order to do something like the Manhattan Project, but in much more controlled, peace-time environments. The atmosphere-ignition prior would have been stamped out pretty quickly by later calculations of physicists to the contrary, and after that research would have gotten back to full steam ahead. I think the counterfactual US would have gotten onto the atom bomb in the early 1950s at the absolute latest with the talent they had in an MP-less world. Just with much greater safety protocols, and without the Russians learning of it in such blatant fashion. Our abilities to detect such weapons being developed elsewhere would likely have also stayed far ahead of the Russians. You could easily imagine a situation where the Russians finally create a weapon in 1960 that was almost as powerful as what we had cooked up by 1950.

Then you're more or less back to an old-fashioned deterrence model, with the twist that the Russians don't actually know exactly how powerful the weapons the US has developed are. This is an absolute good: You can always choose to reveal just a lower bound of how powerful your side is, if you think you need to, or you can choose to remain totally cloaked in darkness. If you buy the narrative that the US were "the good guys" (I do!) and wouldn't risk armaggedon just because they had the upper hand, then this seems like it can only make the future arc of the (already shorter) Cold War all the safer.

I am assuming Gorbachev or someone still called this whole circus off around the late 80s-early 90s. Gotta trim the butterfly effect somewhere.

post-rationalism is where all the cool kids are and where the best ideas are at right now. the post rationalists consistently have better predictions and the 'rationalists' are stuck arguing whether chickens suffer more getting factory farmed or chickens cause more suffering eating bugs outside.

they also let SF get run into the ground until their detractors decided to take over.

There's kind of two clusters, one is people who talk about meditation all the time, the other is center-right people who did drugs once. I think the second group showed up because rationalists are not-so-secretly into scientific racism (because they believe anything they see with numbers in it) and they just wanted to hang out with people like that.

There is an interesting atmosphere where it feels like they observed California big tech 1000x engineer types and are trying to cargo cult the way those people behave. I'm not sure what they get out of it.

One point is that when Mowshowitz is dispelling the argument that abuse rates are much higher for homeschooled kids, he (and the counterargument in general) references a study [1] showing that abuse rates for non-homeschooled kids are similarly high: both around 37%. That paper's no good though! Their conclusion is "We estimate that 37.4% of all children experience a child protective services investigation by age 18 years." 37.4%? That's 27m kids! How can CPS run so many investigations? That's 4k investigations a day over 18 years, no holidays or weekends. Nah. Here are some good numbers (that I got to from the bad study, FWIW) [2], they're around 4.2%.

But, more broadly, the worst failing of the US educational system isn't how it treats smart kids, it's how it treats kids for whom it fails. If you're not the 80% of kids who can somehow make it in the school system, you're doomed. Mowshowitz' article is nearly entirely dedicated to how hard it is to liberate your suffering, gifted student from the prison of public education. This is a real problem! I agree it would be good to solve it!

But, it's just not the problem. Again I'm sympathetic to and agree with a lot of the points in the article, but you can really boil it down to "let smart, wealthy parents homeschool their kids without social media scorn". Fine, I guess. No one's stopping you from deleting your account and moving to California. But it's not an efficient use of resources--and it's certainly a terrible political strategy--to focus on such a small fraction of the population, and to be clear this is the absolute nicest way I can characterize these kinds of policy positions. This thing is going nowhere as long as it stays so self-obsessed.

[0]: https://thezvi.substack.com/p/childhood-and-education-9-scho...

[1]: https://pmc.ncbi.nlm.nih.gov/articles/PMC5227926/

[2]: https://acf.gov/sites/default/files/documents/cb/cm2023.pdf

You can convince a lot of people that you've done your homework when the medium is "an extremely blog post with a bunch of studies attached" even if the studies themselves aren't representative of reality.

BTW, this isn't a defensive posture on my part: I am not plugged in enough to even have an opinion on any rationalist community, much less identify as one.

there's only ~3300 counties in the USA.

i'll let you extrapolate how CPS can handle "4000/day". Like, 800 people with my wife's qualifications and caseload is equivalent to 4000/day. there's ~5000 caseworkers in the US per statistia:

> In 2022, there were about 5,036 intake and screening workers in child protective services in the United States. In total, there were about 30,750 people working in child protective services in that year.

my wife's caseload (adults) "floats around fifty."

My misunderstanding then - what are you speaking to? Even reading this comment, I still don't understand.

> 800 people with my wife's qualifications and caseload is equivalent to 4000/day. there's ~5000 caseworkers in the US

I don't know what the number of children in the system is. as i said in the comment you replied to, here. but the average US CPS worker caseload is 69 cases. which is over 300,000 children per year, because there are ~5000 CPS caseworkers in the US.

I was only speaking to "how do they 'run' that many investigations?" as if it's impossible. I pointed out it's possible with ~1000 caseworkers.

[0]: https://pmc.ncbi.nlm.nih.gov/articles/PMC5087599/

also i wasn't considering "confirmed maltreatment" - just the fact that 4k/day isn't "impossible"

Maybe, but this sounds like some ideologically opposed groups slandering each other to get the moral high ground to me. The papers linked show a pretty typical racialized pattern of CPS calls (Blacks high, Asians low, Whites and Latinos somewhere between) that maybe contraindicates this, for example.

> also i wasn't considering "confirmed maltreatment" - just the fact that 4k/day isn't "impossible"

Yup I think you're right here. I think there's something fuzzy happening with conflating "CPS investigation" with "abuse", but I'm not sure where the homeschool abuse rate comes from.

Predominantly "black" schools receive less funding in general (per student over $2000 less), and as such, need all the student-age people in class. So a "black" family removing their child(ren) from school becomes a fiscal issue, coupled to racial issues, coupled to history; like >60% of "black" children live in a 'single parent household' due to "no man about the house" policies dating back to the 1960s, just as a single example.

I am quoting "black" because i am sensitive to this, and if i had started out with ADOS or NBA/FBA (native black american, foundational black american) i just assume it'd brook argument.

To wrap this all up - "more testing equals more cases."

The whole reason smart people are engaging in this debate in the first place is that professional educators keep trying to train their sights on smart wealthy parents homeschooling their kids.

By the way, this small fraction of the population is responsible for the driving the bulk of R&D.

Rationalism is an ideal, yet those who label themselves as such do not realize their base of knowledge could be wrong.

They lack an understanding of epistemology and it gives them confidence. I wonder if these 'rationalists' are all under age 40, they havent seen themselves fooled yet.

It has a priesthood that speaks for god (quantum). It has ideals passed down from on high. It has presuppositions about how the universe functions which must not be questioned. And it's filled with people happy that they are the chosen ones and they feel sorry for everyone that isn't enlightened like they are.

In the OPs article, I had to chuckle a little when they started the whole thing off by mentioning how other Rationalists recognized them as a physicist (they aren't). Then they proceeded to talk about "quantum cloning theory".

Therein is the problem. A bunch of people vociferously speaking outside their expertise confidently and being taken seriously by others.

Do you have specific examples in mind? (And not to put too fine a point on it, do you think there's a chance that you might be wrong about this assertion? You've expressed it very confidently...)

On "considering what should be the baseline assumption":

https://www.lesswrong.com/w/epistemology

https://www.lesswrong.com/w/priors, particularly https://www.lesswrong.com/posts/hNqte2p48nqKux3wS/trapped-pr...

On the idea that "rationalists think that they can just apply rationality infinitely to everything":

https://www.lesswrong.com/w/bounded-rationality

On the critique that rationalists are blind to the fact that "reason isn't the only thing that's important", generously reworded as "reason has to be grounded in a set of human values", some of the most philosophically coherent stuff I see on the internet is from LW:

https://www.lesswrong.com/w/metaethics-sequence

https://www.lesswrong.com/w/human-values

On "systematically plan to validate":

https://www.lesswrong.com/w/rationality-verification

https://www.lesswrong.com/w/making-beliefs-pay-rent

On "what could hold true for one moment could easily shift":

https://www.lesswrong.com/w/black-swans

https://www.lesswrong.com/w/distributional-shifts

https://www.lesswrong.com/w/forecasting-and-prediction

I support anyone trying to form rational pictures of the universe and humanity. If the LessWrong community approach seems to make sense and is enriching to your understanding of the world then I am happy for you. But, every time I try to take a serious delve into LessWrong, and I have done it multiple times over the years, it sets off my cult/scam alerts.

Kinda like Mensa?

I’m so glad I didn’t join because being around the types of adults that make being smart their identity surely would have had some corrosive effects

However I'm always surprised how much some people want to talk about intelligence. I mean, it's the common ground of the group in this case, but still.

In my opinion, there can’t be a meaningful distinction made between rational and irrational without Popper.

Popper injects an epistemic humility that Bayesianism, taken alone, can miss.

I think that aligns well with your observation.

Most of Popper's key points are elaborated on at length in blog posts on LessWrong. Perhaps they got something wrong?

https://www.yudkowsky.net/rational/bayes

These are the kind of statements I’m referring to. Happy to be falsified btw :) that’s how we learn.

Also note that Popper never called his theory falsificationism.

> On the other hand, Popper’s idea that there is only falsification and no such thing as confirmation turns out to be incorrect. Bayes’ Theorem shows that falsification is very strong evidence compared to confirmation, but falsification is still probabilistic in nature; it is not governed by fundamentally different rules from confirmation, as Popper argued.

Popper's idea that confirmatory evidence has no value at all is obviously implausible. Some obviously implausible things turn out to be true anyway, since the universe is not constrained by our imaginations, but not this one; as the page clearly shows, we know that this particular implausible result is slightly false, and we can use Bayesian probability to calculate exactly how wrong.

I've answered your question, because I think that's what a bare minimum level of courtesy demands, but you keep evading mine. What epistemological propositions of Popper's do you think they're missing?

One example for all. It was claimed that a great rationalist policy is to distribute treated mosquito nets to 3rd-world-ers to help eradicate malaria. On the ground, the same nets were commonly used for fishing and other activities, polluting the environment with insecticides. Unfortunately, rationalists forgot to ask people that live with mosquitos what they would do with such nets.

Could you recommend an article to learn more about this?

Bayesianism requires you to assume / formalize your prior belief about the subject under investigation and updates it given some data, resulting in a posterior belief distribution. It thus does not have the clear distinctions of frequentism, but that can also be considered an advantage.

[1] https://web.mit.edu/hackl/www/lab/turkshop/readings/gigerenz...

Popperians claim that positive justification is impossible.

Popperians claim.Induction doesn't exist (or at least , matter in science)

Popper was prepared to consider the existence of Propensities objective.probabilities, whereas Bayesians, particularly those who follow Jaynes believe in determinism and subjective probability.

Popperian refutation is all or nothing, whereas Bayesian negative information is gradual.

In Popperism, there can be more than one front running or most favoured theory, even after the falsifiable ones have been falsified, since there aren't quantifiable degrees of confirmation.

For Popper and Deutsch, theories need to be explanatory, not just predictive. Bayesian confirmation and disconfirmation only target prediction directly -- if they are achieving explanation or ontological correspondence , that would be the result of a convenient coincidence.

For Popperians, the construction of good theoretical conjectures is as important as testing them. Bayesian seem quite uninterested in where hypotheses come from.

For Deutschians, being hard-to-vary is the preferred principle of parsimony. For Yudkuwsians, it's computation complexity.

Error correction as something you actually do. Popperians like to put forward hypotheses that are easy to refute. Bayesians approve theoretically of "updating", but dislike objections and criticisms in practice.

(Long term) prediction is basically impossible . More Deutsch than Popper -- DD believed that the growth and unpredictability of knowledge . The creation of knowledge is so unpredictable and radical that long term predictions cannot be made. Often summarised to "prediction is impossible". Of course , Bayesians are all about prediction --but the predictive power of Ates tends only to be demonstrated in you models, where the ontology isn't changing under your feet. Their AI I predictions are explicitly intuition based.

Optimism versus Doom. Deutsch is highly optimistic that continuing knowledge creation will change the world for the better (a kind of moral realism is a component of this). Yudkowsky thinks advanced AI is our last invention and will kill us all.*

https://www.lesswrong.com/posts/85mfawamKdxzzaPeK/any-good-c...

I personally don't have that much of an interest in this topic, so I can't critique them for quality myself, but they may at least be of relevance to you.

The doomer utilitarian arguments often seem to involve some sort of infinity or really large numbers (much like EAs) which result in various kinds of philosophical mugging.

In particular, the doomer plans invariably result in some need for draconian centralised control. Some kind of body or system that can tell everyone what to do with (of course) doomers in charge.

“If X, then surely Y will follow! It’s a slippery slope! We can’t allow X!”

They call out the name of the fallacy they are committing BY NAME and think that it somehow supports their conclusion?

Rationalists, mostly self-identified.

> how can they estimate the probability of a future event based on zero priors and a total lack of scientific evidence?

As best as they can, because at the end of the day you still need to make decisions (You can of course choose to do nothing and ignore the risk, but that's not a safe, neutral option). Which means either you treat it as if it had a particular probability, or you waste money and effort doing things in a less effective way. It's like preparing for global warming or floods or hurricanes or what have you - yes, the error bars are wide, but at the end of the day you take the best estimate you can and get on with it, because anything else is worse.

Which is to say that you've made an estimate that the probability is, IDK, <5%, <1%, or some such.

Eliezer did once state his intentions to build "friendly AI", but seems to have been thwarted by his first order reasoning about how AI decision theory should work being more important to him than building something that actually did work, even when others figured out the latter bit.

Instead, unless there's a single winner, we will probably see the knowledge on how to train big LLMs and make them perform well diffuse throughout a large pool of AI researchers, with the hardware to train models reasonably close to the SotA becoming more quite accessible.

I think the people who will benefit will be the owners of ordinary but hard-to-dislodge software firms, maybe those that have a hardware component. Maybe firms like Apple, maybe car manufacturers. Pure software firms might end up having AI assisted programmers as competitors instead, pushing margins down.

This is of course pretty speculative, and it's not reality yet, since firms like Cursor etc. have high valuations, but I think this is what you'd get from the probably pressure if it keeps getting better.

I suspect you'll see a few people "win" or strike it rich with AI, the vast majority will simply be left with a big bill.

It turns out that boom-and-bust capitalism isn’t great for building something that needs to evolve over centuries.

Perhaps American AI efforts will one day be viewed similarly. “Yeah, they had an early rush, lots of innovation, high valuations, and robber barons competing. Today it’s just stale old infra despite the high-energy start.”

https://www.worldatlas.com/articles/highest-railway-cargo-tr...

For what it's worth, I like traveling by train and do so whenever I can, but I'm an outlier. Most Americans look at the travel times and laugh at the premise of choosing a train over a plane. And when I say they look at the travel times, I don't mean they actually bother to look up train routes. They just know that airplanes are several times faster. Delays suffered by trains never get factored into the decision because trains aren't taken seriously in the first place.

Humans are also freight, of course. It is not like the rail companies really care about what kind of fright is on the trains, so long as it is what the customer considers most important (read: most profitable). Humans are deprioritized exactly because they aren't considered important by the customer, which is to say that the customer, who is also the freight in this case, doesn't really want to be on a train in the first place. The customer would absolutely ensure priority (read: pay more, making it known that they are priority) if they wanted to be there.

I understand the train geeks on the internet find it hard to believe that not everyone loves trains like they do, but the harsh reality is that the average American Joe prefers other means of transportation. Should that change in the future, the rail network will quickly accommodate. It has before!

You are comparing USA today to the robber baron phase, whose to say China isn't in the same phase? Lots of money being thrown at new railroads and you have Chinese leaders and best and management leaders chasing that money. When happens when it goes low budget/maintenance mode?

Nonsense. The US has the largest freight rail system in the world, and is considered to have the most efficient rail system in the world to go along with it.

There isn't much in the way of passenger service, granted, but that's because people in the US aren't, well, poor. They can afford better transportation options.

> It turns out that boom-and-bust capitalism isn’t great for building something that needs to evolve over centuries.

It initially built out the passenger rail just fine, but then evolution saw better options come along. Passenger rail disappeared because it no longer served a purpose. It is not like, say, Japan where the median household income is approaching half that of Mississippi and they hold on to rail because that's what is affordable.

This is so misguided view... Trains (when done right) aren't "for the poor", they are great transportation option, that beats both airplanes and cars. In Poland, which isn't even close to the best, you can travel between big cities with speeds above 200km/h, and you can use regional rail for your daily commute, both those options being very comfortable and convenient, much more convenient than traveling by car.

What gives you the idea that rail would be preferable to flying for the NYC to LAS route if only it existed? Even as the crow flies it is approximately 4,000 km, meaning that at 200 km/h you are still looking at around 20 hours of travel in an ideal case. Instead of just 5 hours by plane. If you're poor an additional 15 hours wasted might not mean much, but when time is valuable?

Why would you constrain the route to within a specific state? In fact, right now a high-speed rail line is being planned between Las Vegas and LA.

But outside of Nevada, there are many equivalent distance routes in the US between major population centers, including:

Chicago/Detroit

Dallas/Houston

LA/SF

Atlanta/Charlotte

Passenger train service between Chicago and Detroit already runs 110 MPH (180 KPH) for 90% of the route, using essentially the same trainset that Brightline plans to use. They’ve got a rail tunnel underneath the Detroit River and had been planning to make the route be Chicago/Detroit/Toronto with Customs/Immigration happening on-board the train as it crosses the border. Ontario needs a lot of rail upgrades to make the Canadian leg run at those same speeds or higher.

They’ve made a lot of investments since the 1990s. It’s much improved, though perhaps not as nice as during the golden years when it was a big part of the New York Central system (from the 1890s to the 1960s they had daily trains that went Boston/NYC/Buffalo/Detroit/Chicago through Canada from Niagara Falls to Windsor).

During the first Trump administration, Amtrak announced a route that would go Chicago/Detroit/Toronto/Montreal/Quebec City using that same rail tunnel underneath the Detroit River. It was supposed to start by 2030. We’ll see if it happens.

Right now and since 1979!

I'll grant you that people love to plan, but it turns out that they don't love putting on their boots and picking up a shovel nearly as much.

> But outside of Nevada, there are many equivalent distance routes in the US between major population centers, including

And there is nothing stopping those lines from being built other than the lack of will to do it. As before, the will doesn't exist because better options exist.

There is no magic in this world like you seem to want to pretended. All of those things simply boil down to people. Property rights only exist because people say they do, environmental reviews only exist because people say they do, skilled workers are, well, literally people, and the necessary capital is already created. If the capital is being directed to other purposes, it is only because people decided those purposes are more important. All of this can change if the people want it to.

> HN users sometimes have this weird fantasy that with enough political will it's possible to make enormous changes but that's simply not how things operate in a republic with a dual sovereignty system.

Hell, the republic and dual sovereignty system itself only exists because that's what people have decided upon. Believe it or not, it wasn't bestowed upon by some mythical genie in the sky. The people can change it all on a whim if the will is there.

The will isn't there of course, as there is no reason for the will to be there given that there are better options anyway, but if the will was there it'd be done already.

I've taken a Chinese train from Zhengzhou, in central China, to Shenzhen, and it was fantastic. Cheap, smooth, fast, lots of legroom, easy to get on and off or walk around to the dining car. And, there's a thing where boiling hot water is available, so everyone brings instant noodle packs of every variety to eat on the train.

Can't even imagine what the US would be like if we had that kind of thing.

Getting to the airport in most major cities takes an hour, and then there's the whole pre-flight security theatre, and the flights themselves are rarely pleasant. To add insult to injury, in the US it's usually a $50 cab ride to the airport and there are $28 ham-and-cheese sandwiches in the terminal if you get hungry.

In China and Japan the trains are centrally located, getting on takes ten minutes, and the rides are extremely comfortable. If such a thing existed in the US I think it would be extremely popular. Even if it was just SF-LA-Vegas.

Do you mean beginning in the same city? If so, that's downright hilarious. I live 50 miles clear of the city, out in the middle of nowhere, and can be to the airport in the city in less than an hour.

> If such a thing existed in the US I think it would be extremely popular.

I don't know, if people willingly spend an hour getting from one point to another in the same city, the aren't apt to be concerned about how they get from one city to another. I expect they don't put much thought into anything.

Anyway, New York to Las Vegas spans most of the US. There are plenty of routes in the US where rail would make sense. Between Boston, New Haven, New York City, Philadelphia, Baltimore, and Washington, D.C. Which has the Amtrak Acela. Or perhaps Miami to Orlando. Which has a privately funded high speed rail connection called Brightline that runs at 200 km/h who's ridership was triple what had been expected at launch.

I am, thankfully, not.

> Which has a privately funded high speed rail connection called Brightline that runs at 200 km/h

Which proves that when the will is there, it will be done. The only impediment in other places is simply the people not wanting it. If they wanted it, it would already be there.

The US has been here before. It built out a pretty good, even great, passenger rail network a couple of centuries ago when the people wanted it. It eventually died out simply because the people didn't want it anymore.

If they want it again in the future, it will return. But as for the moment...

Since nobody really wants passenger rail in the US, they don't put in the effort to see that it exists (outside of some particular routes where they do want it). In many other countries, people do want board access to passenger rail (because that's all they can afford), so they put in the effort to have it.

~200 years ago the US did want passenger rail, they put in the work to realize it, and it did have a pretty good passenger rail network at the time given the period. But, again, better technology came along, so people stopped maintaining/improving what was there. They could do it again if they wanted to... But they don't.

1. Freight is easier to manage and has better economics on a dedicated network. The US freight network is extremely efficient as others have pointed out. Other networks, e.g., Germany, instead prioritized passenger service. In Germany rail moves a small proportion of freight (19%) compared to trucks. [0] It's really noticeable on the Autobahn and unlike the US where a lot of truck traffic is intermodal loads.

2. The US could have better rail service by investing in passenger networks. Instead we have boondoggles like the California high-speed rail project which has already burned through 10s of billions of dollars with no end in sight. [1] Or the New Jersey Transit system which I had the pleasure to ride on earlier today to Newark Airport. It has pretty good coverage but needs investment.

[0] https://dhl-freight-connections.com/en/trends/global-freight...

[1] https://en.wikipedia.org/wiki/California_High-Speed_Rail

How so?

> The US freight network is extremely efficient as others have pointed out.

'Others' being literally the comment you replied to.

> The US could have better rail service by investing in passenger networks.

Everything there is can be improved, of course, but to what significance here?

https://www.mlex.com/mlex/antitrust/articles/2355294/us-rail...

Boeing tries to compete with Airbus, where the engineers make maybe 70% of what they do in the US, and has upon that made some really bad decisions. Many aircraft companies are profitable.

The problem is the railroads were purchased by the winners. Who turned out to be the existing winners. Who then went on to continue to win.

On the one hand, I guess that's just life here in reality.

On the other, man, reality sucks sometimes.

Imagine if they were bought by losers.

It's not social media. It's a model the capitalists train and own. Best the rest of us will have access to are open source ones. It's like the difference between trying to go into court backed by google searches as opposed to Lexis/Nexis. You're gonna have a bad day with the judge.

Here's hoping the open source stuff gets trained on quality data rather than reddit and 4chan. Given how the courts are leaning on copyright, and lack of vetted data outside copyright holder remit, I'm not sanguine about the chances of parity long term.

Anything that can't be self-improving or superhuman almost certainly isn't worthy of the moniker "AI". A true AI will be born into a world that has already unlocked the principles of intelligence. Humans in that world would be capable themselves of improving AI (slowly), but the AI itself will (presumably) run on silicon and be a quick thinker. It will be able to self-improve, rapidly at first, and then more rapidly as its increased intelligence allows for even quicker rates of improvement. And if not superhuman initially, it would soon become so.

We don't even have anything resembling real AI at the moment. Generative models are probably some blind alley.

I think that the OP's point was that it doesn't matter whether it's "real AI" or not. Even if it's just a glorified auto-correct system, it's one that has the clear potential to overturn our information/communication systems and our assumptions about individuals' economic value.

That's going to be a swift kick to your economy, no matter how strong.

I recently had an LLM write a function for me that, for a given RGB color value and another integer n > 1, returned to me a list of n RGB colors equidistantly and sequentially spaced around the color wheel starting at the specified RGB value.

For a given system I'm creating, I might have lots of such tasks. That collection of tasks is something that did matter and took some education and skill to complete well.

In the pre-LLM world - assuming I was too busy to handle all the tasks myself - I would have delegated them to a junior software engineer.

In a post-LLM world, I just ask the LLM to implement tasks like that, and I review the code for correctness.

That seems like a pretty transformational change to me, and not just some kind of "rot" being removed from the process.

In my view, rationalists are often "Bayesian" in that they are constantly looking for updates to their model. Consider that the default approach for most humans is to believe a variety of things and to feel indignant if someone holds differing views (the adage never discuss religion or politics). If one adopts the perspective that their own views might be wrong, one must find a balance between confidently acting on a belief and being open to the belief being overturned or debunked (by experience, by argument, etc.).

Most rationalists I've met enjoy the process of updating or discarding beliefs in favor of ones they consider more correct. But to be fair to one's own prior attempts at rationality, one should try reasonably hard to defend one's current beliefs so that they can be fully and soundly replaced if necessary, without leaving any doubt that they were insufficiently supported, etc.

To many people (the kind of people who never discuss religion or politics) all this is very uncomfortable and reveals that rationalists are egotistical and lacking in humility. Nothing could be further from the truth. It takes tremendous humility to assume that one's own beliefs are quite possibly wrong. The very name of Eliezer's blog "Less Wrong" makes this humility quite clear. Scott Alexander is also very open with his priors and known biases / foci, and I view his writing as primarily focusing on big picture epistemological patterns that most people end up overlooking because most people are busy, etc.

One final note about the AI-dystopianism common among rationalists -- we really don't know yet what the outcome will be. I personally am a big fan of AI, but we as humans do not remotely understand the social/linguistic/memetic environment well enough to know for sure how AI will impact our society and culture. My guess is that it will amplify rather than mitigate differences in innate intelligence in humans, but that's a tangent.

I think to some, the rationalist movement feels like historical "logical positivist" movements that were reductionist and socially darwinian. While it is obvious to me that the rationalist movement is nothing of the sort, some people view the word "rationalist" as itself full of the implication that self-proclaimed rationalists consider themselves superior at reasoning. In fact they simply employ a heuristic for considering their own rationality over time and attempting to maximize it -- this includes listening to "gut feelings" and hunches, etc,. in case you didn't realize.

If you want to see how human and tribal rationalists are, go criticize the movement as an outsider. Or try to write a mildly critical NYT piece about them and watch how they react.

It's rather different for a community to say that's a standard they aspire to, which is a lot less ridiculously grandstanding of a position IMO.

>The quantum physicist who’s always getting into arguments on the Internet, and who’s essentially always right

“Guy Who Is Always Right” as a role in a social group is a terrible target, yet it somehow seems like what rationalists are aiming for every time I read any of their blog posts

These folks have a bunch of money because we allowed them to privatize the commons of 20th century R&D mostly funded by the DoD and done at places like Bell Labs, Thiel and others saw that their interests had become aligned with more traditional arch-Randian goons, and they've captured the levers of power damn near up to the presidency.

This has quite predictably led to a real mess that's getting worse by the day, the economic outlook is bleak, wars are breaking out or intensifying left right and center, and all of this traces a very clear lineage back to allowing a small group of people privatize a bunch of public good.

It was a disaster when it happened in Russia in the 90s and its a disaster now.

Aren't these the people who started the trend of writing things like "epistemic status: mostly speculation" on their blog posts? And writing essays about the dangers of overconfidence? And measuring how often their predictions turn out wrong? And maintaining webpages titled "list of things I was wrong about"?

Are you sure you're not painting this group with an overly-broad brush?

They're responsible for funneling huge amounts of funding away from domain experts (effective altruism in practice means "Oxford math PhD writes a book report about a social sciences problem they've only read about and then defunds all the NGOs").

They're responsible for moving all the AI safety funding away from disparate impact measures to "save us from skynet" fantasies.

When big money got involved, the tone shifted a lot. One phrase that really stuck with me is "exceptional talent". Everyone in EA was suddenly talking about finding, involving, hiring exceptional talent at a time where there was more than enough money going around to give some to us mediocre people as well.

In the case of EA in particular circlejerks lead to idiotic ideas even when paired with rationalist rhetoric, so they bought mansions for team building (how else are you getting exceptional talent), praised crypto (because they are funding the best and brightest) and started caring a lot about shrimp welfare (no one else does).

I think that sentence would be a fair description of certain individuals in the EA community, especially SBF, but that is not the same thing as saying that rationalists don't ever express epistemic uncertainty, when on average they spend more words on that than just about any other group I can think of.

Ah. I guess they are working out ecology through first principles, I guess?

I feel like a lot of the criticism of EA and rationalism does boil down to some kind of general criticism of naivete and entitlement, which... is probably true when applied to lots of people, regardless of whether they espouse these ideas or not.

It's also easier to criticize obviously doomed/misguided efforts at making the world a better place than to think deeply about how many of the pressing modern day problems (environmental issues, extinction, human suffering, etc.) also seem to be completely intractable, when analyzed in terms of the average individual's ability to take action. I think some criticism of EA or rationalism is also a reaction to a creeping unspoken consensus that "things are only going to get worse" in the future.

I think it's that combined with the EA approach to it which is: let's focus on space flight and shrimp welfare. Not sure which side is more in denial about the impending future?

I have no belief any particular individual can do anything about shrimp welfare more than they can about the intractable problems we do face.

I think its a result of its complete denial of and ignorance of politics. Because rationalist and effective altruist movements make a whole lot more sense, if you realize they are talking about deeply social and political issues with all politics removed from it. Its technocrat-ism the poster child of the kind of "there is no alternative" neoliberalism that everyone in the western world was indoctrinated into since the 80s.

Its a fundamental contradiction, we don't need to talk about politics because we already know liberal democracies and free-market capitalism is the best we ever going to achieve, faced with the numerous intractable problems we face that can not possibly be related to liberal democracies and free-market capitalism.

The problem is: How do we talk about any issue the world is facing today without ever challenging or even talking about any of the many assumptions the western liberal democracies are based upon? In other words: the problems we face are structural/systemic, but we are not allowed to talk about the structures/systems. That's how you end up with space flight and shrimp welfare and AGI/ASI catastrophizing taking up 99% of everything these people talk about. It's infantile, impotent liberal escapism more than anything else.

They bought one mansion to host fundraisers with the super-rich, which I believe is an important correction. You might disagree with that reasoning as well, but it's definitely not as described.

As far as I know it's never hosted an impress-the-oligarch fundraiser, which as you say would at least have a logic behind it[1] even if it might seem distasteful.

For a philosophy which started out from the point of view that much of mainstream aid was spent with little thought, it was a bit of an end of Animal Farm moment.

(to their credit, a lot of people who identified as EAs were unhappy. If you drew a Venn diagram of the people that objected, people who sneered at the objections[2] and people who identified as rationalists you might only need two circles though...)

[1]a pretty shaky one considering how easy it is to impress American billionaires with Oxford architecture without going to the expense of operating a nearby mansion as a venue, particularly if you happen to be a charitable movement with strong links to the university... [2]obviously people are only objecting to it for PR purposes because they're no smart enough to realise that capital appreciates and that venues cost money, and definitely not because they've got a pretty good idea how expensive upkeep on little used medieval venues are and how many alternatives exist if you really care about the cost effectiveness of your retreat, especially to charitable movements affiliated with a university...

>As far as I know it's never hosted an impress-the-oligarch fundraiser

As far as I know, they only hosted 3 events there before deciding to sell, so this is low-information.

I’m a bit confused by this one.

Are you saying that no-one who identifies as rationalist sneered at the objections? Because I don’t think that’s true.

Yes! It can be true both that rationalists tend, more than almost any other group, to admit and try to take account of their uncertainty about things they say and that it's fun to dunk on them for being arrogant and always assuming they're 100% right!

Because they we doing so many workshops that buying a building was cheaper than renting all the time.

You may argue that organizing workshops is wrong (and you might be right about that), but once you choose to do them, it makes sense to choose the cheaper option rather than the more expensive one. That's not rationalist rhetoric, that's just basic economy.

And I have definitely encountered "if you just listen to me properly you will understand that I am right, because I have derived my conclusions rationally" in in person interactions.

On the balance I'd rather have some arrogance and willingness to be debated and be wrong, over a timid need to defer to centuries of established thought though. The people I've met in person I've always been happy to hang out with and talk to.

I remember as a child coming to the same "if reality is a deception, at least I must exist to be deceived" conclusion that Descartes did, well before I had heard of Descartes. (I don't think this makes me special, it's just a natural conclusion anyone will reach if they ponder the subject). I think it's harmless for me to discuss that idea in public without someone saying "you need to read Descartes before you can talk about this".

I also find my personal ethics are stronly aligned with what Kant espoused. But most people I talk to are not academic philosophers and have not read Kant, so when I want to explain my morals, I am better off explaining the ideas themselves than talking about Kant, which would be a distraction anyway because I didn't learn them from Kant, we just arrived at the same conclusions. If I'm talking with a philosopher I can just say "I'm a Kantian" as shorthand, but that's really just jargon for people who already know what I'm talking about.

I also think that while it would be unusual for someone to (for example) write a guide to understanding relativity without once mentioning Einstein, it also wouldn't be a fundamental flaw.

(But I agree there's no certainly excuse for someone asserting that they're right because they're rational!)

The problem is less clear in philosophy than mathematics, but it's still there. It's really easy on your own terms to come up with some idea that the collective intelligence has already revealed to be fatally flawed in some undeniable manner, or at the very least, has very powerful arguments against it that an individual may never consider. The ideas that have survived decades, centuries, and even millenia against the collective weight of humanity assaulting them are going to have a certain character that "something someone came up with last week" will lack.

(That said I am quite heterodox in one way, which is that I'm not a big believer in reading primary sources, at least routinely. Personally I think that a lot of the primary sources noticeably lack the refinement and polish added as humanity chews it over and processes it and I prefer mostly pulling from the result of the process, and not from the one person who happened to introduce a particular idea. Such a source may be interesting for other reasons, but not in my opinion for philosophy.)

I'm not sure if this counterpoint generalizes entirely to the original critique, since certainly LessWrongers aren't usually posting about or discussing math as if they've discovered it-- usually substantially more niche topics.

Like mathematical logic (in the intersection of math and philosophy) didn't have that many true predecessors and was developed very far by maybe only 5-10 individuals cumulatively, or information theory was basically established by Claude Shannon and maybe two other guys, or various aspects of convex optimization or Fourier analysis were only developed in the 80s or so, it stands to reason that the AI-related applications of various aspects of philosophy are ripe to be developed now. (By contrast, we don't see, as much, people on LW trying to redo linear algebra from the ground up, nor more "mature" aspects of philosophy.)

(If anything, I think it's more feasible than ever before, also, for a bunch of relative amateurs to non-professionally make real intellectual contributions, like noticeably moreso than 100 or even 20 years ago. That's what increasing the baseline levels of education/wealth/exposure to information was intended to achieve, on some level, isn't it?)

I come from a physics background. We used to (and still) have a ton of physicists who decide to dable in a new field, secure in their knowledge that they are smarter than the people doing it, and that anything worthwhile that has already been thought of they can just rederive ad hoc when needed (economists are the only other group that seems to have this tendency...) [1]. It turned out every time that the people who had spent decades working on, studying, discussing and debating the field in question had actually figured important shit out along the way. They might not have come with the mathematical toolbox that physicists had, and outside perspectives that challenge established thinking to prove itself again can be valuable, but when your goal is to actually understand what's happening in the real world, you can't ignore what's been done.

[1] There even is an xkcd about this:

https://xkcd.com/793/

Suppose a foot race. Choose two runners of equal aptitude and finite existence. Start one at mile 1 and one at mile 100. Who do you think will get farther?

Not to mention, engaging in human community and discourse is a big part of what it means to be human. Knowledge isn't personal or isolated, we build it together. The "first principles people" understand this to the extent that they have even built their own community of like minded explorers, problem is, a big part of this bond is their choice to be willfully ignorant of large swaths of human intellectual development. Not only is this stupid, it also is a great disservice to your forebears, who worked just as hard to come to their conclusions and who have been building up the edifice of science bit by bit. It's completely antithetical to the spirit of scientific endeavor.

Or because western culture reflects this theme continuously through all the culture and media you've immersed in since you were a child?

Also the idea is definitely not new to Descartes, you can find echoes of it going back to Plato, so your idea isn't wrong per se. But I think it underrates the effect to which our philosophical preconceptions are culturally constructed.

All serious works in philosophy (Kant especially) are subject to interpretation. Whole research programmes exist around the works of major philosophers, interpreting and building on their works.

One cannot really do justice to e.g. the Critique of Pure Reason by discussing it based on a high level summary of the “main ideas” contained in it. These works have had a major impact on the history of Western philosophy and were groundbreaking at the time (and still are).

I find a lot of people in software have an insufferable tendency to simply ignore entire bodies of prior art, prior research, etc. outside of maybe computer science (and even that can be rare), and yet they act as though they are the most studied participants in the subject, proudly proclaiming their "genius insights" that are essentially restatements of basic facts in any given field that they would have learned if they just bothered to, you know, actually do research and put aside their egos for half a second to wonder if maybe the eons of human activity prior to their precious existence might have led to some decent knowledge.

https://www.smbc-comics.com/?id=2556

This is a feature, not a bug, for writers who hold an opinion on something and want to rationalize it.

So many of the rationalist posts I've read through the years come from someone who has an opinion or gut feeling about something, but they want it to be seen as something more rigorous. The "first principles" writing style is a license to throw out the existing research on the topic, including contradictory evidence, and construct an all new scaffold around their opinion that makes it look more valid.

I use the "SlimeTimeMoldTime - A Chemical Hunger" blog series as an example because it was so widely shared and endorsed in the rationalist community: https://slimemoldtimemold.com/2021/07/07/a-chemical-hunger-p... It even received a financial grant from Scott Alexander of Astral Codex Ten

Actual experts were discrediting the series from the first blog post and explaining all of the author's errors, but the community soldiered on with it anyway, eventually making the belief that lithium in the water supply was causing the obesity epidemic into a meme within the rationalist community. There's no evidence supporting this and countless take-downs of how the author misinterpreted or cherry-picked data, but because it was written with the rationalist style and given the implicit blessing of a rationalist figurehead it was adopted as ground truth by many for years. People have been waking up to issues with the series for a while now, but at the time it was remarkable how quickly the idea spread as if it was a true, novel discovery.

I think that SlimeMoldTimeMold's rise and fall was actually a pretty big point in favor of the "rationalist community".

That feels like revisionist history to me. It rose to fame in LessWrong and SlateStarCodex, was promoted by Yudkowski, and proliferated for about a year and half before the takedowns finally got traction.

While it was the topic du jour in the rationalist spaces it was very difficult to argue against. I vividly remember how hard it was to convince anyone that SMTM wasn't a good source at the time, because so many people saw Yudkowski endorse it, saw Scott Alexander give it a shout out, and so on.

Now Yudkowski has gone back and edited his old endorsement, it has disappeared from the discourse, and many want to pretend the whole episode never happened.

> (I don't remember any detailed takedowns of SlimeMoldTimeMold coming before that article, but maybe there are).

Exactly my point. It was criticized widely outside of the rationalist community, but the takedowns were all dismissed because they weren't properly rationalist-coded. It finally took someone writing it up in the form of rationalist rhetoric and seeding it into LessWrong to break the spell.

This is the trend with rationalist-centric contrarianism: You have to code your articles with the correct prose, structure, and signs to get uptake in the rationalist community. Once you see it, it's hard to miss.

Do you have any examples of this that predate that LW article? Ideally both the critique and its dismissal but just the critique would be great. The original HN submission had a few comments critiquing it but I didn't see anything in depth (or for that matter as strident).

Don't worry, HN commenters can figure out the truth about Yudkowsky's articles from the first principles. They have already figured out that EAs no longer care about curing malaria, despite https://www.givewell.org/charities/top-charities only being a Google search away.

At the end, they will give you a lecture about how everyone hates people who are smug and talk about things they have no clue about. The lecture will then get a lot of upvotes.

I wish more people who detest rationalist ideology were curious about actual evidence.

Similarly, Aurornis made a claim that "Scott Alexander predicted at least $250 million in damages from Black Lives Matter protests", when if fact (as the very link provided by Aurornis shows) Scott predicted that the probability of such thing happening was 30%, i.e. it's more likely not to happen.

Elsewhere in this thread, another user, tptacek, claims that "Scott Alexander published some of his best-known posts under his own name". When I asked him for evidence, he said "I know more about this than you, and I'm not invested in this discussion enough to educate you adversarially". Translated: no evidence provided.

.

From my perspective, this all kinda proves my point.

Is the rationality community the only place where people care about evidence? Of course not.

But is the rationality community a rare place where people can ask for evidence in an informal debate and reasonably expect to actually get it? Unfortunately, I think the evidence we got here points towards yes.

Hacker News is a website mostly visited by smart people who are curious about many things. They are even smart enough to notice that some claims are suspicious, and ask for evidence. But will they receive it? No, they usually won't.

And in the next debate on the same topic, most likely the same false claims will be made again, maybe by people who have learned them in this thread. And the claims will be upvoted again.

This is an aspect where the rationality community strives to do better. It is not about some people being smarter than others, or whatever accusations are typically made. It is about establishing social norms where people e.g. don't get upvoted for making unsubstantiated negative claims about someone they don't like, without being asked to back it up, or get downvoted.

> Aren't these the people who started the trend of writing things like "epistemic status: mostly speculation" on their blog posts? And writing essays about the dangers of overconfidence? And measuring how often their predictions turn out wrong? And maintaining webpages titled "list of things I was wrong about"?

There's a lot of in-group signaling in rationalist circles like the "epistemic status" taglines, posting predictions, and putting your humility on show.

This has come full-circle, though, and now rationalist writings are generally pre-baked with hedging, both-sides takes, escape hatches, and other writing tricks that make it easier to claim they weren't entirely wrong in the future.

A perfect exaple is the recent "AI 2027" doomsday scenario that predicts a rapid escalation of AI superpowers followed by disaster in only a couple years: https://ai-2027.com/

If you read the backstory and supporting blog posts from the authors they are filled to the brim with hedges and escape hatches. Scott Alexander wrote that it was something like "the 80th percentile of their fast scenario", which means when it fails to come true he can simple say it wasn't actually his median prediction anyway and that they were writing about the fast scenario. I can already predict that the "We were wrong" article will be more about what they got right with a heavy emphasis on the fact that it wasn't their real median prediction anyway.

I think this group relies heavily on the faux-humility and hedging because they've recognized how powerful it is to get people to trust them. Even the comment above is implying that because they say and do these things, they must be immune from the criticism delivered above. That's exactly why they wrap their posts in these signals, before going on to do whatever they were going to do anyway.

If you want to say their humility is not genuine, fine. I'm not sure I agree with it, but you are entitled to that view. But to simultaneously be attacking the same community for not ever showing a sense of maybe being wrong or uncertain, and also for expressing it so often it's become an in-group signal, is just too much cognitive dissonance.

That's my point: Their rhetorical style is interpreted by the in-group as a sort of weird infallibility. Like they've covered both sides and therefore the work is technically correct in all cases. Once they go through the hedging dance, they can put forth the opinion-based point they're trying to make in a very persuasive way, falling back to the hedging in the future if it turns out to be completely wrong.

The writing style looks different depending on where you stand: Reading it in the forward direction makes it feel like the main point is very likely. Reading it in the backward direction you notice the hedging and decide they were also correct. Yet at the time, the rationalist community attaches themselves to the position being pushed.

> But to simultaneously be attacking the same community for not ever showing a sense of maybe being wrong or uncertain, and also for expressing it so often it's become an in-group signal, is just too much cognitive dissonance.

That's a strawman argument. At no point did I "attack the community for not ever showing a sense of maybe being wrong or uncertain".

In most writing, people write less persuasively on topics they have less conviction in.

Ok, let's scroll up the thread. When I refer to "the specific criticism that I quoted", and when you say "implying that because they say and do these things, they must be immune from the criticism delivered above": what do you think was the "criticism delivered above"? Because I thought we were talking about contrarian1234's claim to exactly this "strawman", and you so far have not appeared to not agree with me that this criticism was invalid.

edit: my apologies, that was someone else in the thread. I do feel like between the two comments though there is a "damned if you do, damned if you don't". (The original quote above I found absurd upon reading it.)

“Damned if you do” indeed…

This is right, but doesn't actually cover all the options. It's damned if you [write confidently about something and] do or don't [hedge with a probability or "epistemic status"].

But the other option, which is the one the vast majority of people choose, is to not write confidently about everything.

It's fine, there are far worse sins than writing persuasively about tons of stuff and inevitably getting lots of it wrong. But it's absolutely reasonable to criticize this choice, irregardless of the level of hedging.

They can't really be blamed for the fact that others go on to take the ideas more seriously than they intended.

(If anything, I think that at least in person, most rationalists are far less confident and far less persuasive than the typical person in proportion to the amount of knowledge/expertise/effort they have on a given topic, particularly in a professional setting, and they would all be well-served to do at least a normal human amount of "write and explain persuasively rather than as a mechanical report of the facts as you see them".)

(Also, with all communities there will be the more serious and dedicated core of the people, and then those who sort of cargo-cult or who defer much, or at least some, of their thinking to members with more status. This is sort of unavoidable on multiple levels-- for one, it's quite a reasonable thing to do with the amount of information out there, and for another, communities are always comprised of people with varying levels of seriousness, sincere people and grifters, careful thinkers and less careful thinkers, etc. (see mobs-geeks-sociopaths))

(Obviously even with these caveats there are exceptions to this statement, because society is complex and something about propaganda and consequentialism.)

Alternately, I wonder if you think there might be a better way of "writing unconfidently", like, other than not writing at all.

My point wasn't to nit-pick individual predictions, it was a general explanation of how the game is played.

Since Scott Alexander comes up a lot, a few randomly selected predictions that didn't come true:

- He predicted at least $250 million in damages from Black Lives Matter protests.

- He predicted Andrew Yang would win the 2021 NYC mayoral race with 80% certainty (he came in 4th place)

- He gave a 70% chance to Vitamin D being generally recognized as a good COVID treatment

This is just random samples from the first blog post that popped in Google: https://www.astralcodexten.com/p/grading-my-2021-predictions

It's also noteworthy to read that a lot of his predictions are about his personal life, his own blogging actions, or [redacted] things. These all get mixed in with a small number of geopolitical, economic, and medical predictions with the net result of bringing his overall accuracy up.

(I feel like you're probably getting upvotes from people who feel similarly, but sometimes I feel like nobody ever writes "I agree with you" comments, so the impression is that there's only disagreement with some point being made.)

Then you start encountering the weirder parts. For me, it was the group think and hero worship. I just wanted to read interesting takes on new topics, but if you deviated from the popular narrative associated with the heroes (Scott Alexander, Yudkowski, Cowen, Aaronson, etc.) it felt like the community's immune system identified you as an intruder and started attacking.

I think a lot of people get drawn into the idea of it being a community where they finally belong. Especially on Twitter (where the latest iteration is "TPOT") it's extraordinarily clique-ish and defensive. It feels like high school level social dynamics at play, except the players are equipped with deep reserves of rhetoric and seemingly endless free time to dunk on people and send their followers after people who disagree. It's a very weird contrast to the ideals claimed by the community.

Since when is that what we do here? If he'd written that he'd decided to become vegetarian, would we all be out here talking about how vegetarians are so annoying and one of them even spat on my hamburger one time?

And then of these uncalled-for takedowns, several -- including yours -- don't even seem to be engaging in good-faith discourse, and seem happy to pile on to attacks even when they're completely at odds with their own arguments.

I'm sorry to say it but the one who decided to use their free time to leer at people un-provoked over the internet seems to be you.

(Indeed, I think it's in worse faith to try to guilt trip people who are just expressing critical opinions. It's fine - good, even! - to disagree with those people, but this particular comment has a very "how dare you criticize something!" tone that I don't think is constructive.)

I genuinely don't understand how you can point to someone's calibration curve where they've broadly done well, and cherry pick the failed predictions they made, and use this not just to claim that they're making bad predictions but that they're slimy about admitting error. What more could you possibly want from someone than a tally of their prediction record graded against the probability they explicitly assigned to it?

One man's modus ponens, as it goes.

lol, what? That was a civil comment. This seems like an excellent example of the point being made. Replying to a perfectly reasonable but critical comment with "please be civil" is super condescending.

So is stuff like "one man's modus ponens".

Look, we get it, you're talking to people who found this stuff smart and interesting in the past. But we got tired of it. For me, I realized after awhile that the people I most admired in real life were pretty much the opposite of the people I was reading the most on the internet. None of the smartest people I know talk in this snooty online iamsosmart style.

> This isn’t about me being an expert on these topics and getting them exactly right, it’s about me calibrating my ability to tell how much I know about things and how certain I am.

> At least $250 million in damage from BLM protests this year: 30%

Aurornis:

> I forgot about the slightly condescending, lecturing tone that comes out when you disagree with rationalist figureheads.

> Since Scott Alexander comes up a lot, a few randomly selected predictions that didn't come true:

> He predicted at least $250 million in damages from Black Lives Matter protests.

Is this a "perfectly reasonable but critical comment"?

Am I condescending if I say that predicting a 30% chance that something happens means predicting a 70% chance that it won't happen... so the fact that it didn't happen probably shouldn't be used as "gotcha!"?

(I did waffle upon re-reading my comment and thinking it could have been more civil. But then decided that this person is also being very thin skinned. So I think you're right that we're both right.)

How was it condescending or lecturing?

> You could simply ask "Can you provide examples" instead of the "If you ____ then I suggest ____" form.

Why is that not equally condescending or lecturing?

It just isn't! An inability to accurately identify what comes off as condescending is kind of the point...

"Are you being condescending" is a subjective judgement that other people will make up their own minds about. You can't control what people think about things you say and do, and they aren't "oppressing" you by making up their own minds about that.

You seem to be trying to insinuate that Alexander et. al. are pretending to know how things will turn out and then hiding behind probabilities when they don't turn out that way. This is missing the point completely. The point is that when Alexander assigns an 80% probability to many different outcomes, about 80% of them should occur, and it should not be clear to anyone (including Alexander) ahead of time which 80%.

> He predicted at least $250 million in damages from Black Lives Matter protests.

Many sources estimated damages at $2 billion or more (see https://www.usatoday.com/story/news/factcheck/2022/02/22/fac... and links from there), so this did in fact come true.

Edit: I see that the prediction relates to 2021 specificially. In the wake of 2020, I think it was perfectly reasonable to make such a prediction at that confidence level, even if it didn't actually turn out that way.

> He predicted Andrew Yang would win the 2021 NYC mayoral race with 80% certainty (he came in 4th place)

> He gave a 70% chance to Vitamin D being generally recognized as a good COVID treatment

If you make many predictions at 70-80% confidence, as he does, you should expect 20-30% of them not to come true. It would in fact be a failure (underconfidence) if they all came true. You are in fact citing a blog post that is exactly about a self-assessment of those confidence levels.

Also, he gave a 70% chance to Vitamin D not being generally recognized as a good COVID treatment.

> These all get mixed in with a small number of geopolitical, economic, and medical predictions with the net result of bringing his overall accuracy up.

The point is not "overall accuracy", but overall calibration - i.e., whether his assigned probabilities end up making sense and being statistically validated.

You have done nothing to establish that any correlation between the category of prediction and his accuracy on them.

Yes that's the point, these people hedge like crazy to the point that they say nothing, they mean nothing, and effectively predict nothing.

> He predicted at least $250 million in damages from Black Lives Matter protests.

He says

> 5. At least $250 million in damage from BLM protests this year: 30%

which, by my reading means he assigns it greater-than-even odds that _less_ than $250 million dollars in damages happened (I have no understanding of whether or not this result is the case, but my reading of your post suggests that you believe that this was indeed the outcome).

You say > He gave a 70% chance to Vitamin D being generally recognized as a good COVID treatment

while he says > Vitamin D is _not_ generally recognized (eg NICE, UpToDate) as effective COVID treatment: 70% (emphasis mine)

"Aren't these the people who"...

> And writing essays about the dangers of overconfidence? And measuring how often their predictions turn out wrong? And maintaining webpages titled "list of things I was wrong about"?

What's the value of that if it doesn't appear to be reasonably put to their own ideas. What you described otherwise is just another form of the exact kind of self-congratulation often (reasonably, IMO) lobbed at these "people"

Note they are a mostly American phenomenon. To me, that's a consequence of the oppressive culture of "cliques" in American schools. I would even suppose it is a second-order effect of the deep racism of American culture: the first level is to belong to the "whites" or the "blacks", but when it is not enough, you have to create your own subgroup with its identity, pride, conferences... To make yourself even more betterer than the others.

I see this in rationalist spaces too – it doesn't really make sense for people to talk about things that they believe in strongly but that 95%+ of the public also believe in (like the existence of air), or that they don't have a strong opinion on.

I am a very vocal doomer on AI because I predict with high probability it's going to be very bad for humanity and this is an opinion which, although shared by some, is quite controversial and probably only held by 30% of the public. Given the importance of the subject, my confidence, and that fact I feel the vast majority of people are even wrong or are significantly underweighting caetrosphohic risks, I have to be vocal about it.

Do I acknowledge I might be wrong? Sure, but for me the probability is low enough that I'm comfortable making very strong and unqualified statements about what I believe will happen. I suspect others in the rationalist community like Eliezer Yudkowsky think similarly.

Also, when you say you have a strong belief, does that mean you have emptied you retirement accounts and you are enjoying all you can in the moment until the end comes?

For example, I won't cross the street without 99.99% confidence that I will survive. I cross streets so many times that a lower threshold like 99% would look like insanely risky dart-into-traffic behaviour.

If an asteroid is heading for earth, then even a 25% probability of apocalyptic collision is enough that I would call it very high, and spend almost all my focus attempting to prevent that outcome. But I wouldn't empty my retirement account for the sake of hedonism because there's still a 75% chance I make it through and need to plan my retirement.

There is a term for this. "Getting stuck up your own butt." It wouldn't be so bad except that said people often take on an air of absolute superiority because they used "only logic" and in their head they can not be wrong. Many people end up thinking like this as teenagers or 20 somethings, but most will have someone in their life who smacks them over the head and tells them to stop being so foolish, but if you have enough money and the Internet you can insulate yourself from that kind of oversight.

People in these communities are generally quite smart, and it’s seductive to reason in a purely logical, deductive way. There is real value in thinking rigorously and in making sure you’re not beholden to commonly held beliefs. But, like you said, reality is complex, and it’s really hard to pick initial premises that capture everything relevant. The insane conclusions they get to could be avoided by re-checking & revising premises, especially when the argument is going in a direction that clashes with history, real-world experience, or basic common sense.

The whole reason we even have time to think this way is because we are at the peak of an industrial civilization that has created a level of abundance that allows a lot of people a lot of time to think. But the whole situation that we live in is not stable at all, "progress" could continue, or we could hit a peak and regress. As much as we can see a lot of long-term trajectories (eg. peak oil, global warming), we really have no idea what will be the triggers and inflection points that change the social fabric in ways that are unforeseeable and quickly invalidate whatever prior assumptions all that deep thinking was resting upon. I mean 50 years ago we thought overpopulation was the biggest risk, and that thinking has completely flipped even without a major trajectory change for industrial civilization in that time.

The real conflict here is between Darwinism and enlightenment ideals. But I have yet to see any self-styled Rationalists take this seriously.

Emotionally I don’t subscribe to this view. Rationally I do.

My critique for rational people is that they don’t seem to fully take experience into account. It’s assumptions + rationality + experience/data + whatever strong inclinations one has that seems to be the full picture for me.

That always seemed like a meaningless argument to me. To an outside observer free will is indistinguishable from a random process over some range of possibilities. You aren’t going to randomly go to sleep with your hand in a fire, there’s some hard coded biology preventing that choice but that only means human behavior isn’t completely random, hardly a groundbreaking discovery.

At the other end we have no issues making an arbitrary decision where there’s no way to predict what the better choice is. So what exactly does free will bring to the table that we’re missing without it? Some sort of mystical soul, well what if that’s also deterministic? Unpredictability is useful in game theory, but computers can get that from a hardware RNG based on quantum processes like radioactive decay, so it doesn’t mean much.

Finally, subjectively the answer isn’t clear so what difference does it make?

Same as that is not the lived experience. I notice that I care about free choice.

The idea that there's no free will may be a pessimistic outlook to some but to me it's a strictly neutral one. It used to be a bit negative, until I looked more closely that there's a difference between looking at a situation objectively and having a lived experience. When it comes to my inclinations and how I want to live life, lived experience takes precedence.

I don't have my thoughts sharp on it, but I don't think the concept even exists philosophically, but I think that's also what you're getting at. It's a conceptual remnant from the past.

But though that is the colloquial meaning, it doesn't line up with what people say they want: you want to make your choice according to your own reasons. You want free choice. But unless your own reasoning includes a literal throw of the dice, your justifications deterministically decide the outcome.

"Free will" is the ability to make your own choices, and for most people most of the time, those choices are deterministic given the options and knowledge available. Free will and determinism are not only compatible, but necessarily so. If your choices weren't deterministic, it wouldn't be free will.

But when you probe people, while a lot of people will argue in ways that a philosopher might call compatibilist, my experience is that people will also strongly resist the notion that the only options are randomness and determinism. A lot of people have what boils down to a religious belief in a third category that is not merely a combination of those two, but infuses some mysterious third options where they "choose" that they can't explain.

Most of the time, people who believe there is no free will (and can't be), like me, take positions similar to what you described, that - again - a proponent of free will might describe as compatibilist, but sometimes we oppose the term for the reason above: A lot of people genuinely believe in a "third option" for choices are made.

And so there are really two separate debates on free will: Does the "third option" exist or not, and does "compatibilist free will" exist or not. I don't think I've ever met anyone who seriously disagrees that "free will" the way compatibilists define it exists, so when compatibilists get into arguments over this, it's almost always a misunderstanding...

But I have met plenty of people who disagree with the notion that things are deterministic "from the outside".

Approaching this subject from a rational perspective divorces you from subject and makes it impossible to perceive. You have to immerse yourself in it and one way to do that is magical practice. Having direct experience of the universe responding to your actions and mindset eventually makes it absurdly clear that the universe bears intelligence and it's in this intelligence that free will operates.

I'd never thought before now to connect magic this directly to free will. Thanks for the opportunity to think this through! If you're interested in a deeper discussion, happy to jump on a call.

From the outside, this is indistinguishable from randomness. But from the inside, the difference is that the individual had a say in what the action would be.

Where this tends to get tangled up with notions of a soul, I think, is that one could argue that such a free choice needs some kind of internal state. If not, then the grounds by which the person makes the choice is a combination of something that is fixed and their environment, which then intuitively seems to reduce the free-will process to a combination of determined and random. So the natural thing to do is then to assign the required "being-ness" (or internal state if you will) to a soul.

But there may exist subtle philosophical arguments that sidestep this dilemma. I am not a philosopher: this is just my impression of what commonsense notions of free will mean.

Since there's no free will, outcomes are determined by luck, and what matters is how lucky we can make people through pit-of-success environments. Rust makes people luckier than C++ does.

I also have much less patience for blame than I do in a world with free will. I believe, for example, that blameless postmortems lead to much better outcomes than trying to pretend people had free will to make mistakes, and therefore blaming them for those mistakes.

You can get to these positions through means other than rejection of free will, but the most robust grounds for them are fundamentally deterministic.

Such system often deal with uncertainty quite well including random noise on their inputs. The output ends up a function of both logic and randomness, but can still be useful.

What I'm saying is that there's no logical point to the concept "should" unless you have some concept of free will: everything that happens must happen, or is entirely random.

This is not correct. Whether or not you have free will, stuff influences and is influenced by other stuff, so these arguments are not meaningless or worthless.

> This choice was either made 13 billion years ago at the Big Bang, or it is an entirely random process.

I had thought of this before, and what I had decided is that both of these are also independent of having free will. For example, if the initial state includes unknown and uncomputable transcendental numbers which can somehow "encode" free will and then the working of physics is deterministic, then it is still possible (although not necessarily mandatory) to have free will, even though it is deterministic.

E.g. punishment for the sake of retribution is near impossible to morally justify if you don't believe in free will because it means you're punishing someone for something the had no agency over.

Similarly, wealth disparities can't be excused by someone choosing to work harder, because they had no agency in the "decision".

You can still justify some degree of punishment and reward, but a lack of free will changes which justifications are reasonable very substantially.

E.g. punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals, and that has emotionally difficult consequences. For example, for non-premeditated murders carried out out of passion rather than e.g. gang crimes, the odds of someone carrying out another is extremely low and the odds that the fear of a long prison sentence is an actual deterrence is generally low, and and so long prison terms are hard justifiable once vengeance is off the table.

And so holding on to a belief in free will is easier to a lot of people than the alternative.

My experience is that there are few issues where people so easily get angry than if you suggest we don't have free will once they start thinking through the consequences (and some imagined ones...).

For example

> E.g. punishment for the sake of retribution is near impossible to morally justify if you don't believe in free will because it means you're punishing someone for something the had no agency over.

and

> E.g. punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals

are simply logical to me (even without assuming any lack of free will).

So what is emotionally difficult about this, as you claim?

I'm saying it's emotionally difficult to people because I've had this discussion many times over then last 30+ years and I've seen first hand how most people I have this conversation with tend to get angry and agitated over the prospect of not having moral cover for vengeance.

I live in Germany.

When I observe the whole societal and political situation in the USA from the outside, it seems to me that it is rather "two blocks where in each of these there is somewhat an internal consensus regarding a quite some political positions. On the other hand, each of these two blocks is actively fighting the other one."

On the other hand, for Germany, I would claim that the opinions in society rather consist of "lots of very diverse stances (though in contrary to the USA less pronounced on the extreme ends) on a lot of topics that make it hard to reach a larger set of followers or some consensus in a larger group, i.e. in-fighting about all kinds of topics without these positions forming political camps (and the fractions for different opinions can easily change when the topic changes)."

Thus, in the given example, this means for a person out-crying "too short" sentences on social media, you will very likely find one who is out-crying the opposite position.

However, it would seem that not everyone believes that, though.

(It is not quite as simple as it might seem, because the situation is not necessarily always that clear, but other than that, I would agree that it is logical and reasonable, that punishment is only justified from the point of view of reducing offending and reoffending rates and only if it actually achieves those goals.)

But, sure, I personally do not believe in free will. I'm saying there is no rational basis for thinking anyone has free will ever. I'm saying there is no evidence to suggest free will is possible. In fact, I'll go so far as to say that believing in free will is a religious belief with no support.

But that doesn't mean that events does not have effects on what happens next, just that we don't have agency. That an IF ... THEN ... ELSE ... statement is purely deterministic for deterministic inputs does not mean that changing the inputs won't affect the outputs.

If you "choose" to lay down and do nothing because you decide nothing matters because you don't have free will, you will still lose your job and starve. That it wasn't a "true" "free" choice does not change the fact that it has consequences.

One of the consequences of coming to accept that free will is an illusion is that you need to come to terms with what that means for your beliefs about a wide range of things.

Including that vengeance which might seem moral to some extent if the person who did something to you or others had agency suddenly become immoral. But we still have the feelings and impulses. Reconciling that is hard for a lot of people, and so a lot of people in my experience when faced with a claim like the one I made above that we have no free will tend to react emotionally to the idea of the consequences of it.

If there are non deterministic processes that can be proven to exist, and those interact with deterministic processes, doesn't it follow that the deterministic process becomes non deterministic (since the result of the interaction is necessarily non deterministic), and that it is not continually deterministic.

So - can you see how nothing can be deterministic other than in isolation (or thought experiment really)?

Edit0: typo

We can’t measure things to arbitrary precision due to quantum mechanics, but Philosophy isn’t bound by the actual physical universe we inhabit. Arbitrary physical models allow for the possibility of infinite precision in measurement and calculation resulting in perfect prediction of future states forever. Alternatively, you could have a universe of finite precision (think Minecraft) which also allows for perfect calculation of all future states from initial starting concepts.

Not certain that philosophy is not bound by our universe - is that something you could elaborate (or lend a link) on?

To apply these hypotheticals to our universe implies (from my understanding) that the density of information present at any and all times since it's inception was present (while compressed) at it's creation/whatever - which I imagine I can find some proof of theoretical maximum information density and information compression compare that to the first state of the universe we can measure to have a better idea if it tracks.

I simply mean it’s happy to assume perfect information, perfect clones, etc. The trolly problem generally ignores the possibility that choosing a different track could with some probability result in derailment because the inherent question is simplified by design. We don’t need for the possibility of perfect cloning to exist to consider the ramifications of such etc.

I guess that's the point of any hypothetical, exploring a simplified model of something complex, but it's not easy to simplify the fabric of reality itself.

In a world without free will crimes of passion are simply the result of the situation which means that person would always chose murder in that situation. People who would respond with murder in an unacceptably wide range of situations is an edge case worth consideration without free will. Alternatively if we want nobody to respond with murder in a crime of passion situation evolutionary pressure could eventually work even without free will.

> E.g. punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals, and that has emotionally difficult consequences. For example, for non-premeditated murders carried out out of passion rather than e.g. gang crimes, the odds of someone carrying out another is extremely low and the odds that the fear of a long prison sentence is an actual deterrence is generally low, and and so long prison terms are hard to justify once vengeance is off the table.

That’s assuming absolute certainty about what happened. Punishments may make sense as a logical argument even if it’s only useful in a subset of cases if you can’t be absolutely sure which case something happened to be.

Uncertainty does a lot to align emotional heuristics and logical actions.

> In a world without free will crimes of passion are simply the result of the situation which means that person would always chose murder in that situation. People who would respond with murder in an unacceptably wide range of situations is an edge case worth consideration without free will.

This is a significant argument. However, there is also worth considering if that is actually accurate, and if such a situation will occur (in a case where whoever would be killed would not effectively protect themself from this).

> That’s assuming absolute certainty about what happened. Punishments may make sense as a logical argument even if it’s only useful in a subset of cases if you can’t be absolutely sure which case something happened to be.

It is true that you do not have absolute certainty, but neither should you arrest someone who is not guilty.

> Uncertainty does a lot to align emotional heuristics and logical actions.

In some cases, yes, but it is not always valid. But, even if it is, this does not mean that you should not consider it logically if you are able to do so.

False, the punisher also has no will, so it doesn’t matter.

Whether or not you have a choice and free will, you can influence and be influenced by other stuff, since that is how anything is doing.

> punishment might still be justified from the point of view of reducing offending and reoffending rates, but if that is the goal then it is only justified to the extent that it actually achieves those goals

I do agree with that, and I think that whether or not you have free will is not significant. Being emotionally difficult is not what makes it good or bad in this case (and it also does not seem to be so emotionally difficult to me, anyways). Reducing reoffending rates is what is important.

(Another issue is knowing if they are actually guilty (you shouldn't arrest people who are not actually guilty of murder); this is not always certain, either.)

I also think that it should mean that prisoners should not be treated badly and that prison sentences should not be too long. (Also, they shouldn't take up too much space by the prisons, since they should have free space for natural lands and for other buildings and purposes, but that is not quite the same issue, though.)

However, there may be cases where a fine might be appropriate, in order to pay for damages (although if someone else is willing to forgive them then such a fine may not be required). This does not justify a prison sentence or stuff like that, though.

Also, some people will just not like them anymore if they are accused of murder, even if they are not put in prison and not fined. This is not the issue for police and legal things; it is just what it will be. And, if it becomes known, people who disagree with the risk assessment can try to avoid someone.

And, if someone does commit a crime again and may have opportunity to do again in future, then this can be considered as being wrong the first time and this time hopefully you can know better.

What do you mean by that? It still exists doesn't it? Albeit in a probabilistic sense that becomes non-probabilistic at larger scales.

I don't know much about quantum other than the high level conceptual stuff.

It's controversial, but here is the argument that the answer is "no": See https://flownet.com/ron/QM.pdf

Or if you prefer a video: https://www.youtube.com/watch?v=dEaecUuEqfc

>Under QIT, a measurement is just the propagation of a mutually entangled state to a large number of particles.

eyeroll so it's MWI in disguise, but MWI is quantum realism. Illusion they talk about is that the observed macroscopic state is a part of the bigger superposition (incomplete observation). But that's dumb, even if it's a part of a bigger state, it's still real, because it's not made up, but observed.

That's kind of like saying that GRW is Copenhagen in disguise. It's not wrong, but only because it's making the word "disguise" do some pretty heavy lifting.

> MWI is quantum realism

No, it isn't because it can't account for the Born rule. See:

https://blog.rongarret.info/2019/07/the-trouble-with-many-wo...

(That is apparently the definition the author of the linked article uses, guessing by his reaction: "Wait, what??? There is no 'well defined notion of how many branches there are?'")

I can only say that I have never met a proponent of MWI who meant this.

> I can only say that I have never met a proponent of MWI who meant this.

What can I say? There are a lot of MWI proponents who profess to believe this. Here, for example, is Sean Carroll answering the question, "How many parallel universes are there?"

https://www.youtube.com/watch?v=7tQiy5iCX4o

Of course, he doesn't actually give a concrete answer, but he very strongly implies that the question has an answer, i.e. that the question is a meaningful one to ask, and that implies that the MWI does in fact mean that there is a discrete number of clearly separated worlds.

In fact, I challenge you find a single example of a prominent MWI proponent saying something in public (which is to say, in a public forum or a publication whose target audience is the general public) that even implies that the many-worlds of the MWI are not distinct, countable entities. I only know of one example, and it is very well hidden.

There is a more fundamental problem: if the MWI does not mean "a discrete number of clearly separated worlds" then it fails as an interpretation of QM, i.e. as a solution to the measurement problem. The whole point is that measurements appear to produce discrete outcomes despite the fact that the math says that everything is one big quantum superposition. If all you have to say about this is, "Yeah, it's all one big quantum superposition" then you have failed to solve the problem. You have simply swept the hard part under the rug.

In the video, Sean Carroll talks to a non-expert audience, so he must simplify some things, and then it is your or my guess about what the unsimplified version was supposed to be. He says something like: "we don't know, even whether it is finite or infinite, but if it is finite it is a very large number such as 10^10^123". But notice that he also uses as an analogy an interval from 0 to 1, which can be split to half as many times as you need.

You see this as him believing in discrete separated universes, of which there is a definite number (potentially infinite). Yes, that makes sense.

I see another possible understanding, that he is talking about "meaningfully different" universes, because that is what we care about on the macro level. To explain what I mean, imagine that we observe two particles. Any of them can be in a huge number of possible positions, moving in a huge number of possible directions, at a huge number of possible speed. But if we ask whether those two particles hit each other and transformed into another particle, that kinda collapses this huge possibility space into a "yes / no" question. Out of practically infinity, two meaningfully different options.

On a macro level, either the cat is alive or it is dead. Those are two meaningfully different states. If we focus on one particle in the cat's body, there is a continuum of where precisely that particle could be, and what momentum it has. So from the particle's perspective, there is a continuum of options. But from the cat's perspective, and the cat's owner's perspective, this continuum does not matter; unless it changes the macro state, i.e. the particle kills the cat, or at least maybe hits its neuron and makes it do something differently. So it seems possible to me that Sean Carroll talks about the number of worlds that are different from human perspective.

Then there is another problem in physics that we don't know how/whether the very space and time are quantized. We use the mathematical abstraction of a "real number" that has an infinite number of digits after the decimal dot, but of course that infinite number of digits can never be observed experimentally. We don't know. Maybe it is something like what Wolfram says, that on a deep level, spacetime is a discrete graph evolving according to some rules. If something like that would be the case, that would reduce the possible number of states in the universe, even on the micro level, to a huge but finite number. And the mixed state of the multiverse would consist of this finite number of branches, each of them assigned a tiny complex amplitude. So that's another way how things could get finite.

And I am saying this just as a random guy who never studied these things, I just sometimes read something on the topic, and some ideas feel to me like obvious consequences of the stuff that is "in the water supply". So I believe that if I see a solution to a problem, then if it makes sense, someone like Sean Carroll is 10000x more likely to notice the problem and the solution, and develop it much further than I ever could. Or when you make a survey, and a half or a third of people who study quantum physics for living say that some version of MWI seems like the correct interpretation to them, I don't believe there is a simple devastating argument against it that all of these people have simply missed.

OK, well, let me tell you as a non-random guy who has studied these things extensively that the MWI is very commonly misrepresented. It is not a case of simplification for a lay audience, it is flat-out lying, at least most of the time. The math does not say that there are parallel universes. All the math tells you is that in order to recover the results of experiments you have to throw away some of the information contained in the wave function. MWI proponents interpret this by saying that the discarded information has to correspond to something real, and they call that thing "parallel universes". But there are three problems with this. First, the MWI does not explain the Born rule. Second, the math doesn't tell you whether or not the discarded parts of the wave function describe something real or not. It is possible that mathematical operation of discarding parts of the wave function actually corresponds to real physical phenomenon, i.e. that whatever is described by the discarded parts of the wave function actually ceases to exist. This is a tenable scientific hypothesis. It's not easy to actually make it work, but it can be done and has been done. It's called GRW collapse [1]. So anyone who tells you that the MWI is the only possible scientifically tenable interpretation of QM is lying. And anyone who leaves open even the possibility that the "parallel universes" contained in the wave function are discrete is also lying. The only MWI proponent I've ever seen being intellectually honest about this.David Deutsch in his book "The Beginning of Infinity" chapter 11.

The third problem with the MWI is something called the "preferred basis problem". This one is harder to describe succinctly, and some people claim it has been solved, but I don't agree with them. In a nutshell, all two-state QM experiments rely on some macroscopic apparatus to split a particle into a superposition of two states. But if you model the entire universe as a quantum system, this apparatus is itself a quantum system that can be in a superposition of states, so you can't say, "The polarizing beam splitter is aligned vertically or it is aligned horizontally" any more than you can say "the cat is alive or it is dead" without begging the question.

---

[1] https://en.wikipedia.org/wiki/Ghirardi%E2%80%93Rimini%E2%80%...

The math tells that there are no privileged parts of wave function.

>So anyone who tells you that the MWI is the only possible scientifically tenable interpretation of QM is lying.

Didn't you admit yourself that if MWI works it's a big deal and will kick the chair from under other interpretations?

That's true. But my senses tell me that there is a privileged part of the wave function, namely, the branch that I'm in.

The way I think about it nowadays is that QM is like a Necker cube. You can look at it in two different ways. You can take the God's-eye view and look at the entire wave function, or you can take the mortal's eye view and look at only a proper subset of the wave function (which is necessary in order to recover classical reality). But you can't do both at the same time. For my day-to-day life, I have no choice but to take the mortal's-eye view because I am a mortal. All of the things that matter to me depend on classical reality, and so depend on my suspension of disbelief and acting as if my branch of the multiverse is privileged, even if I can intellectually jump out of the system momentarily and recognize that the mortal's eye view is necessarily incomplete.

> Didn't you admit yourself that if MWI works it's a big deal and will kick the chair from under other interpretations?

That depends on what you mean by "works". If someone can derive the Born rule from the Schrodinger equation that will be a big deal, a slam-dunk Nobel prize. But no one has done it, and I'm pretty sure it can't be done. I'm pretty sure that the Born rule is an emergent property of our branch of the multiverse. I believe the same is true of the Second Law and even three-dimensional space. You can slice-and-dice the wave function to give you physical spaces with any number of dimensions, but three is the magic number that gives you atoms and stars and planets with stable orbits [1] and so on. So I'm pretty sure the Born rule can only be explained by the anthropic principle. There's probably a Nobel prize waiting for the person who turns that intuition into a theorem.

---

[1] https://physics.stackexchange.com/questions/50142/gravity-in...

You could also sense that your location is privileged, because the observable universe is neatly centered at it, but science will prioritize Copernican principle over your senses.

>But you can't do both at the same time.

This doesn't match what you do. Tracing extracts mortal's-eye view from God's-eye view, so in God's-eye view you have both.

>depend on my suspension of disbelief and acting as if my branch of the multiverse is privileged

If your branch exists, it's sufficient for your day to day life, there's not much else to disbelieve. There's no need for it to be privileged. Do you worry that Earth isn't more privileged than Mars?

>But no one has done it, and I'm pretty sure it can't be done.

But quantum physics doesn't allow it. It's quantitative science where all observed phenomena are computable. If they aren't computable, then quantum physics doesn't predict them and thus diverges from observation. And Schrödinger equation is how predictions are made, collapse and measurement only act on what already exists before them and don't create anything new. So if Born rule is an observed phenomenon, it must be computable from Schrödinger equation. Also if Born rule holds with certainty, then it's a pure state, and observation won't do anything to it, so Born rule can't be created by observation.

>There's probably a Nobel prize waiting for the person who turns that intuition into a theorem.

This was argued by Max Tegmark https://arxiv.org/abs/gr-qc/9702052 I thought it's a famous diagram https://en.wikipedia.org/wiki/File:Spacetime_dimensionality....

Yes, that's right. This is not an easy problem to solve. This is why it took thousands of years for mankind to realize that the earth is not at the center of the universe. The difference between that and the MWI is that there is actual evidence against geocentrism. There is no evidence against my-branch-centrism. Not only that, but the theory itself predicts that there cannot possibly be any such evidence. So the MWI is self-defeating. The only way there could be evidence for it is if it's wrong.

> in God's-eye view you have both

Nope. The mortal's-eye view is fundamentally incompatible with the god's-eye view. This is the reason that the measurement problem is a thing in the first place.

> if Born rule is an observed phenomenon, it must be computable from Schrödinger equation

Only if the SE is a complete description of reality, and it manifestly is not.

(If you want to argue that the Born rule is not "an observed phenomenon" then I don't know what to tell you. Maybe go hang out with the flat-earthers and lunar landing denialists. You may find kindred spirits there.)

> This was argued by Max Tegmark

Yes, the 3-D space part. That is old news. It's the Born Rule that (AFAIK) no one has yet derived.

One branch interpretations are based on geocentric prejudice that the observer's state isn't changed much by observation (because observer doesn't feel change), and when the observer's state doesn't change much, we get geocentrism. But mathematics of quantum physics shows otherwise: the observer's state suffers decoherence and splits into macroscopic superposition, which is a big change and thus debunks assumption of unchanged observer's state. When observer's state changes significantly, observation becomes subject to relativity effect just like in case of spinning Earth.

>The only way there could be evidence for it is if it's wrong.

And what it means when there's no such evidence?

>The mortal's-eye view is fundamentally incompatible with the god's-eye view.

But then tracing must be fundamentally unable to extract mortal's-eye view from god's-eye view. What you say doesn't match what you do.

>If you want to argue that the Born rule is not "an observed phenomenon" then I don't know what to tell you.

I argue that Born rule is an observed phenomenon, and all observed phenomena are purely quantitative physical processes computable from Schrödinger equation, Born rule is the same, otherwise quantum physics wouldn't predict observation of Born rule.

Formally you might need measurement, but the trick is to convert the given problem into a problem of certainty, then measurement is trivial, and prediction is completely calculated from Schrödinger equation. Coincidentally Born rule is such a certain fact, so it doesn't matter if you measure it or not, measurement doesn't do much to certain facts, it's sufficient if you only calculate this certain fact and leave it as is without measuring it.

It's not just that the observer doesn't feel change, it is that no experiment can demonstrate this change, not even in principle.

> tracing must be fundamentally unable to extract mortal's-eye view from god's-eye view

Why? Because that is manifestly not the case.

> Born rule is the same

Again, you are manifestly wrong. If someone had figured out how to derive the BR from the SE it would be Big News [1].

---

[1] https://blog.rongarret.info/2024/04/the-scientific-method-pa...

The only interpretation that does not have this problem is the NCI because in that interpretation I am part of the fundamental ontology, at least some of the time.

Well, now I see that QIT isn't quite there. You say classical behavior emerges by tracing, mathematically, not as a physical process? In MWI classical behavior emerges as a physical process, not by tracing. That "look at part of the system (in which case you see classical behavior)" is provided by linear independence of different branches, so each observer naturally observes their branch from inside, and it looks isolated from other branches.

Huh??? No, of course not. The Born rule is about probabilities. It cannot manifest in a single measurement.

> classical behavior emerges by tracing, mathematically, not as a physical process?

No. The mathematical description of classical outcomes emerges by tracing, which is to say, by discarding information. The physical interpretation of that is left open.

> In MWI classical behavior emerges as a physical process

That's right. MWI commits to a physical interpretation of the math. But there is no scientific or philosophical justification for this, and in fact, when you dig into the details all kinds of problems emerge that are swept under the rug by its proponents. Nonetheless, many MWI proponents insist that it is the One True Interpretation, including some who really ought to know better.

> each observer naturally observes their branch from inside, and it looks isolated from other branches.

Yes, I know. But this doesn't solve the problem. In order to get a mathematical description of me I have to trace the wave function in my preferred basis, which is to say, I have to throw out all of the other branches. And this is not just a computational hack. It's mathematically necessary. Discarding information is the only way to get classical, irreversible processes (like measurement) out of the unitary dynamics of the wave function. So a reasonable interpretation of the math is that I exist only if parallel universes don't. And I'm pretty sure I exist.

I'm not telling you this because I expect you to accept it, merely to show you that the MWI is not self-evidently the One True Interpretation.

---

[1] https://blog.rongarret.info/2009/04/on-shadow-photons-and-re...

(Note that I wrote this 16 years ago, so not everything is 100% accurate, but I stand by the central point.)

Other branches can be demonstrated indirectly by 1) quantitatively verifying unitary dynamics, 2) indirectly observing branching, 3) demonstrating that other theories are wrong. Branches are just superposition, if you want to eliminate branches, you should eliminate superposition with pilot wave or superdeterminism or something like that. This kind of unobservability isn't unique to MWI, in general theory of relativity we observe only a part of the universe, the rest being beyond event horizon and is unobservable. Do you believe only observable part of the universe exists and beyond it nothing exists?

Huh??? When?

> Other branches can be demonstrated

Sure, that's just QM 101. What you cannot demonstrate experimentally, not even in principle, is the existence of other branches with different macroscopic configurations than our own. Such branches are IPUs.

In the blog post you linked above:

>No, it isn't because it can't account for the Born rule. See:

>https://blog.rongarret.info/2019/07/the-trouble-with-many-wo...

>What you cannot demonstrate experimentally, not even in principle

I provided 3 ways to demonstrate it experimentally, even in principle, not sure what problem you have.

Again, huh??? Where in that blog post do I try to "get this [sic] statistics from one measurement"?

> I provided 3 ways to demonstrate it experimentally, even in principle, not sure what problem you have.

No, you didn't. You apparently don't understand what is meant by "branches with different macroscopic configurations than our own" and I don't have time to explain it to you. Sorry. Go read up on decoherence, and then come back and describe an experiment that can demonstrate the existence of a fully decohered branch. You can't, because if you could it would by definition not be fully docohered.

In the discussion how different people place bets on A and B outcomes of experiment. Well, you didn't state clearly why you believe that MWI doesn't account for Born rule. MWI accounts for Born rule as statistics of measurements, and the discussion of bets is the closest this in that blog post to consideration of statistics of measurements, but that discussion seemingly considers one measurement, that's why it doesn't see statistics.

>Go read up on decoherence, and then come back and describe an experiment that can demonstrate the existence of a fully decohered branch.

It looks like a logical problem to me. You suggest that decoherence both produces and doesn't produce fully decohered branches? Violation of the law of excluded middle? If the law of excluded middle doesn't work, I don't think experiments can demonstrate anything.

That's what the whole post was about. The MWI doesn't account for the Born rule unless you add additional, questionable assumptions like branch indifference to the SE.

> You suggest that decoherence both produces and doesn't produce fully decohered branches?

No, that is not even remotely what I am saying. You are beginning to sound like a troll.

Then you don't understand quantum mechanics at all. You should read this:

https://flownet.com/ron/QM.pdf

The TL;DR is that measurement and entanglement are the same phenomenon. A particle can become entangled with a detector even if the detector doesn't register anything.

But that is neither here nor there. Why do you get interference with no detectors? Your theory is that a detector at one slit is somehow paired with a "virtual detector" in a parallel branch at the other slit. But why would that "virtual detector" go away when the real detector is removed? Why is it never the case that there is a "virtual detector" at either slit unless there is a real detector at one of them?

When you remove detector and start next measurement, you start with your one branch, branches from previous measurements don't affect it, the phenomenon happens during decoherence, nothing happens after it.

Your article explains this with branching without saying the word.

Because that is the best explanation for what I observe.

> by declaring that sense data do not reflect reality, you've cut yourself off form the possibility of knowing reality altogether

That is true, but only in the uninteresting sense that I can never completely eliminate the possibility that I am living in the Matrix. So yes, it's possible that I'm wrong about the existence of objective reality. But if objective reality is itself an illusion, it's a sufficiently compelling illusion that I'm not going to go far wrong by acting as if it were real.

That seems squishy, as what constitutes "going far wrong" is not meaningful under skeptical assumptions.

A better stance is one of cognitive optimism that avoids the irrationality of skepticism. Skepticism is irrational, because it leads to incoherence, and because there is no rational warrant to categorically doubt the senses. For doubt to be rational, there must be a reason for it. To doubt without reason is not to be rational, but to be willful, and willful beliefs cannot be reasoned with; they are not the product of evidence or inference — and they certainly aren't self-evident — but rather the product of arbitrary choice. The logical possibility of living in the Matrix is no reason for doubting the senses, just as the logical possibility of there being poison in your sandwich is no reason for doubting you'll survive eating it.

The difference between our positions is that I begin from a position of natural trust toward the senses and toward reason as the only rational possibility and default. I have no choice but to reason well or to reason poorly. I recognize that my senses and my inferences can err, but it does not follow that they always err. Indeed, the very claim that they can err presumes I can tell when they do.

So, if my inferences lead me to a position that undermines their own coherence, then I must conclude that my inferences are wrong (including those that led me to adopt a certain interpretation of, say, scientific measurements).

> Because that is the best explanation for what I observe.

But if your explanation involves contradiction of what you observe, then that is not only not the best explanation, but no explanation at all! An explanation cannot deny the thing it seeks to explain. Thus, by denying the objective reality of what you perceive, you are barred from inferring that denial.

I can be more precise about this. It means that the predictions I make on the basis of this assumption are very likely to be correct.

> Skepticism is irrational

No, it isn't. The vast majority of my beliefs about the world are not a result of direct observations, but nth-hand accounts. I believe, for example, that the orbit of Mercury precesses, but not because I've ever measured it myself, but rather because I heard it from a source that I consider credible. But assessing the credibility of a source is hard and error-prone, especially nowadays. There is always the possibility that a source is mistaken or actively trying to deceive you. And even for things you observe first-hand there are all kinds of cognitive biases you have to take into account. So skepticism is warranted.

> I begin from a position of natural trust toward the senses

That will lead you astray because your senses are unreliable.

> if your explanation involves contradiction of what you observe

But it doesn't. At worst it involves a contradiction of what I think I observe.

Or for a tldr look for the three body problem or try to find a solution to a double pendulum!

Objects without free will aren’t able to come to conclusions like this.

https://www.lesswrong.com/s/kNANcHLNtJt5qeuSS

"Moloch hasn't won" is a lengthy critique of the argument you are making here.

The fact that you can write this sentence, consider it to be true, and yet still hold in your head the idea that the future might be bad but it's still important to have children suggests that "contact with reality" is not a curse.

Why can't that observation be taken into account? Isn't the entire point of the approach accounting for all inputs to the extent possible?

I think you are making invalid assumptions about the motivations or goals or internal state or etc of the actors which you are then conflating with the approach itself. That there are certain conditions under which the approach is not an optimal strategy does not imply that it is never competitive under any.

The observation is then that rationalism requires certain prerequisites before it can reliably out compete other approaches. That seems reasonable enough when you consider that a fruit fly is unlikely to be able to successfully employ higher level reasoning as a survival strategy.

Of course it can be. I'm saying that AFAICT it generally isn't.

> rationalism requires certain prerequisites before it can reliably out compete other approaches

Yes. And one of those, IMHO, is explicit recognition that rationalism does not triumph simply because it is rational, and coming up with strategies to compensate. But the rationalist community seems too hung up on things like malicious AI and Roko's basilisk to put much effort into that.

I'm sympathetic to the idea that we know nothing because of the reproductive impulse to avoid doing or thinking about things that led our ancestors to avoid procreation, but such a conclusion can't be total because otherwise it is self defeating because is is contingent on rationalist assumptions about the mind's capacity to model knowledge.

Even then that might not always be the case. Sometimes there are severe time or bandwidth or energy or other constraints that preclude carefully collecting data and thinking things through. In those cases a heuristic that is very obviously not derived from any sort of critical thought process might well be the winning strategy.

There will also be cases where the answer provided by the rational approach will be to conform to some other framework. For example where cult type ingroup dynamics are involved across a large portion of the population.

Exactly right. It is not rationalism per se that is the problem, it is the way that The Rationalists are implementing it, the things they are choosing to focus their attention on. They are worried about things like hostile AI and Roko's Basilisk when what they should be worried about is MAGA, because that is not being driven by rationalism, it is being driven by Christian nationalism. MAGA is busily (and openly!) undermining every last hint of rationalism in the U.S. government, but the Rationalist community seems oddly unconcerned with this. Many self-styled Rationalists are even Trump supporters.

I'm not sure I entirely understand what you're arguing here, but I absolutely do agree that the most powerful force in the universe is natural selection.

The term "survival of the fittest" predates Darwin's Origin of Species, and was adopted by Darwin within his lifetime, btw.

If gender equality and intellectual achievement don't produce children, then that isn't "darwinism selecting rationality out". You can't expect the continued existence of finite lifespan organisms if there are no replacement organisms. Raising children is hard work. The people who believe in gender equality and intellectual achievement made the decision to not want more of themselves, particularly when their belief in gender equality entails not wanting male offspring. The alternative is essentially freeloading and expecting others, who do not share the beliefs, to produce children for you and also to teach them the "enlightened" belief of forcing "enlightened" beliefs onto others (note the circularity, the initial conditions are usually irrelevant and often just a fig leaf to perpetuate the status quo).

I never said there was. Darwin said it because he didn't know anything about genes, but that mistake was corrected by Dawkins.

> If gender equality and intellectual achievement don't produce children, then that isn't "darwinism selecting rationality out".

Why not?

> The people who believe in gender equality and intellectual achievement made the decision to not want more of themselves

That's the wrong way to look at it. Individuals are not the unit of reproduction. Genes are. Genes that build brains that want to have (and raise) children are more likely to propagate than genes that build brains that don't, all else being equial. So it is not rationality per se that is the problem -- rationality can provide a reproductive advantage because it lets you, for example, build technology. The problem is that non-rational brains can parasitically benefit from the phenotype of rational brains, at least for a while. But in the long run this is not a stable equilibrium.

I didn't say it was. I said that the Rationalist community is not taking the implications of Darwinism into account when they choose where to focus their attention. This is what leads them to fixate on hostile AI and the MWI when what they should be worried about is the rise of MAGA. But not only is that not what they are worried about, many self-styled Rationalists are Trump supporters.

The fact that humans are intelligent at all and Enlightenment peoples currently dominate the world suggests otherwise.

Huh??? How do you figure? AFAICT the world is dominated by Donald Trump, Xi Jinping, and Vladimir Putin (if you reckon by power) or Christians and Muslims (if you reckon by population). None of these individuals or groups can be properly categorized as "Enlightenment peoples", certainly not with a capital E.

https://en.wikipedia.org/wiki/Age_of_Enlightenment

I guess that depends on what you consider "the world". It makes no sense to even talk about the West dominating "the world" before 1492. The first truly global Western empire was Britain, but it was also the last. It was replaced by the U.S. but it was never really global. Even at the height of its power after WW2 the USSR was a credible rival. After the fall of the USSR in 1991 the U.S. was the sole undisputed superpower for a little while, but that came to an abrupt end on September 11, 2001 and the subsequent wars in Afghanistan and Iraq.

I think you are over-extrapolating the past into the future. The mindset and culture that produced U.S. hegemony in the 20th century seems to me to be mostly extinct. The U.S. was indeed ruled by rationalism (more or less) from the time of its founding through the mid-to-late 20th century, but there is precious little of that left today. Certainly the power structure in the U.S. today is actively hostile to rationalism, and I don't see a whole lot of rationalism in play in the opposition either.

I'm sure you would be able to predict what a rationalise will say when you ask them what future they prefer: one where we maximises for the number of humans or one with fewer humans but better lives

That depends on what you mean by "follow". You have to "follow" Darwinian evolution for the same reason you have to "follow", say, the law of gravity. That doesn't mean you can't build airplanes and spacecraft, but you still have to acknowledge and "follow" the law of gravity. You can't just glue feathers to your arms, jump off a cliff, and hope for the best. (Actually, rationalists aren't even gluing feathers to their arms. They are doing the equivalent of jumping off a cliff because they just don't believe gravity applies to them.)

[UPDATE]

> unnatural sex is wrong

The problem with that argument is that homosexuality is not unnatural. Many, many species have homosexual relations. Accounting for this is a little bit challenging, but the fact is undeniable.

https://en.wikipedia.org/wiki/Homosexual_behavior_in_animals

A counterexample is meiotic drive, where alleles disrupt the meiotic process in order to favour their own transmission, even if the alleles in question ultimately produce a less fit organism.

Whilst this is not an inherently positive observation, I think it does illustrate that the fatalistic picture you're painting here is incorrect. There's room for tentative optimism.

That is not correct. Darwin did make a mistake, but it was not the fundamental dynamics of the process, but that he chose the wrong unit of selection. Darwin thought that selection selected for individuals or species when in fact it selects for genes. Richard Dawkins is the person who figured this out, but Darwin knew nothing about genes (OoS was published only three years after Gregor Mendel's work) so he still gets the credit nothwithsanding this mistake.

Of course. Variation is the other part.

> gene flow or genetic drift

Those are two mechanisms by which variation occurs. This is not something Darwin got wrong, he just didn't have all the data. Genes were unknown in Darwin's time.

Clearly it isn't Darwinian selection, because it can favour transmission of genes which are not conducive to survival.

Clearly it isn't variation, because it tends to reduce the degree of genetic polymorphism within a population.

> it isn't Darwinian selection, because it can favour transmission of genes which are not conducive to survival

This is the one thing that Darwin got wrong: he thought that the unit of selection was the organism, but it's not. The unit of selection is the gene. Richard Dawkins is the one who figured this out. Meiotic drive is not Darinian selection, but it is selection. But Darwin could not possibly have known this because genes were unknown at the time.

You should not interpret that historical success to imply future success as it depended on non-sustainable groundwater extraction.

Eg, https://en.wikipedia.org/wiki/Ogallala_Aquifer

> Many farmers in the Texas High Plains, which rely particularly on groundwater, are now turning away from irrigated agriculture as pumping costs have risen and as they have become aware of the hazards of overpumping.

> Sixty years of intensive farming using huge center-pivot irrigators has emptied parts of the High Plains Aquifer.

> as the water consumption efficiency of the center-pivot irrigator improved over the years, farmers chose to plant more intensively, irrigate more land, and grow thirstier crops rather than reduce water consumption--an example of the Jevons Paradox in practice

How will the Great Plains farmers get water once the remaining groundwater is too expensive to extract?

Salt Lake City cannot simply build desalination plants to fix its water problem.

I expect the bad experiences of Okies during the internal migration of the Dust Bowl will be replicated once the temporary (albeit century-long) relief of using fossil water is exhausted.

I think you only need to look at the water politics of the Great Salt Lake to see the difficulty.

Look at how little water use has changed during the last 25 years of the southwestern North American megadrought.

The US policy appears to be to pray for rain.

Incidentally, the flaw in this theory is in thinking you understand what all the existential risks are.

Suppose you clock "malicious AI" as a huge risk and then hamper AI, but it turns out the bigger risk is not doing space exploration, which AI would have accelerated, because something catastrophic yet already-inevitable is going to happen to the Earth in a few hundred years and if we're not sustainably multi-planetary by then it's all over.

The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

Rationalist community understands that very well. They even know how to put bounds on the unknowns and their own lack of information.

> The thing evolution teaches us is that diversity is a group survival trait. Anybody insisting "nobody anywhere should do X" is more likely to cause an ELE than prevent one.

Right. Good thing they'd agree with you 100% on this.

What's the probability of AI singularity? It has never happened before so you have no priors and any number you assign will be pure speculation.

>And yet people have figured out how to make predictions more narrow than shrugging

And?

> Same is true about anything you're trying to forecast, by definition of it being in the future

There might be some flaws in this line of reasoning...

Singularity obviously never happened before, and if anyone bothered to read up on what they're talking about, they'd realize that no one is trying to predict what happens then, because the singularity is defined as the time when changes accelerate to such a degree that we have no baseline to make any predictions whatsoever.

So when people speculate on when that is, they're trying to forecast the point forecasting breaks; they do it by extrapolating from known examples and trends, to which we do have baselines.

Or, in short: we know how it is to ride an exponent, we just never rode one long enough to fall off of it; predicting singularity is predicting when the exponent gets steep enough we can't follow, which is not unlike predicting any other trend people do. Same methods and caveats apply.

Most of the time we make predictions based on how similar events happened in the past. For completely novel situations it's close to impossible to make a prediction and reckless to base policy on such a prediction.

But it was necessary at the beginning of flight and the flight to the moon would've never been possible either without a few talented people being able to make predictions about scenarios they knew little about.

There are just way too many people around nowadays, which is why most of us never get confronted with such novel topics and consequently we don't know how to reason about it

No they don't. They think they can do this because they've accidentally reinvented the philosophy "logical positivism", which philosophers gave up on because it doesn't work. (This is similar to how they accidentally reinvented reconstructing arguments and called it "steelmanning".)

https://metarationality.com/probability-limitations

There are others, such as the unproven, narcissistic and frankly unlikely-to-be-true assumption that humanity continuing to exist is a net positive in the long run.

This is effectively a religious belief you are espousing.

They also secondarily believe everyone has an IQ which is their DBZ power level; they believe anything they see that has math in it, and IQ is math, so they believe anything they see about IQ. So if you avoid trying to find out your own IQ you can just believe it's really high and then you're going.

Unfortunately this lead them to the conclusion that computers have more IQ than them and so would automatically win any intellectual DBZ laser beam fight against them / enslave them / take over the world.

An actual argument would be that intelligence doesn't work like that. Two people with IQ 100 cooperating together does not produce an IQ 200 solution.

There is the "wisdom of crowds". If a random member of a group is more than 50% likely to be correct, the average of the group is more likely to be correct than its members individually. But that has a few assumptions, for example that each member tries to figure out things independently (as opposed to everyone waiting for the highest-status member to express their opinion, and then agreeing with it -- in that case the entire group is only as smart as the highest-status member).

But you cannot leverage this to simply invite 1000 random people in your group and ask them to invent a Theory of Everything; because the assumption that each member is more than 50% likely to be correct does not apply in this case. So that is one of the limits of people working together.

(And this already conveniently ignores many other problems found in real life, such as conflict of interests, etc.)

IMO it's fine to pick a favorite and devote extra resources to it. But that turns less fine when one also starts working to deprive everything else of any oxygen because it's not your favorite. (And I'm aware that this criticism applies to lots of communities.)

Even if an individual person chooses to direct all their donations to a single cause, there's no way to get everyone to donate to a single cause (nor is EA attempting to). Money gets spread around because people have different values.

It absolutely does take some money away from other causes, but only in the sense that all charities do: if you give a lot to one charity, you may have less money to give to others.

It really annoys me when people say that those religious cultists do that.

They derive their bullshit from faulty, poorly thought out premises.

If you fuck up the very firsts calculations of the algorithm, it doesn't matter how rigorous all the subsequent steps are. There results are going to be all wrong.

I get the feeling these people often want to seem smarter than they are, regardless of how smart they are. And they want to get money to ostensibly "consider these issues", but really they want money for nothing.

If they wanted to do right by the future masses, they should be looking to the things that are affecting us right now. But they treat those issues as if they'll work out in the wash.

The current sums invested and donated in altruist causes are rounding errors themselves compared to GDPs of countries, so the revealed preferences of those investing and donating to altruist causes is to care about the future and the present also.

Are you saying that they should give a greater preference to help those who already exist rather than those who may exist in the future?

I see a lot of Peter Singer’s ideas in modern “effective” altruism, but I get the sense from your comment that you don’t think that they have good reasons for doing what they do, or that their reason leads them to support well-meaning but ineffective solutions. I am trying to understand your position without misrepresenting your point or goals. Are you naysaying or do you have an alternative?

https://en.wikipedia.org/wiki/Peter_Singer

If they wanted to help, they should be focused on the now. Global poverty, climate change, despotic world leaders. They should be aligning themselves against such things.

But instead what we see is essentially not that. Effective altruism is a lot like the Democratic People's Republic of Korea, a bit of a misnomer.

To be pedantic, DPRK is run via the will of the people to a degree comparable to any country. A bigger misnomer is the west calling liberal “democracy”, just democracy.

The elite letting the people choose between a few candidates is not a democracy. There are no “democratic” countries in that way.

Ba da Ching

Don’t forget — the enemies of the west are all bad and evil. And the west is free and fair and democratic and not a scourge on the rest of the world.

It's not arguable, it's simply wrong. But to understand that you would have to understand much more about the DPRK.

> The elite letting the people choose between a few candidates is not a democracy.

It is, but more importantly, your framing is wrong. Every democracy has several levels of democratic institutions (local, state, federal etc.), there are often 'surprise' election winners; there are also non-elective ways for non-elite people to influence policy. DPRK has none of this.

I have never come across some one like you who knows about DPRK outside of what western state depts say and of course they swallow it all up. But people do sometimes lie and say they know so much like you right now

A lot of them argue that poor countries essentially don't matter. Climate change is not an extinction event and there should an authoritarian world government to prevent nuclear conflict to minimize the risk of nuclear extinction.

>In his dissertation On the Overwhelming Importance of Shaping the Far Future (2013), supposedly “one of the best texts on existential risks,”[9] Nicholas Beckstead meditates on the “ripple effects” a human life might have for future generations and concludes “that saving a life in a rich country is substantially more important than saving a life in a poor country” due to the higher level of innovation and economic productivity attained in these countries.[10]

https://umbau.hfg-karlsruhe.de/posts/philosophy-against-the-...

there's something pathologically, virally sick about wealth accumulation's primary function being the further accumulation of wealth. for a movement rooted in "rationalism," EA seems pretty irrationally focused on excusing antisocial behavior.

Then we are lucky that EA promotes giving more to charity as the primary function of accumulation of wealth.

> EA seems pretty irrationally focused on excusing antisocial behavior.

Is a guy getting a well-paid job at Microsoft and donating half of his salary to African charities really your best example of antisocial behavior?

no, we are not lucky. EA-good-because-charity-good is a brain-pretzel way of lobbying against equitable taxation.

> Is a guy getting a well-paid job at Microsoft and donating half of his salary to African charities really your best example of antisocial behavior?

you're inventing a competitive debate regarding a hypothetical "best example of antisocial behavior". i didn't target anyone specifically with any part of my post.

“The purpose of a system is what it does.”

If you assume we eventually figure out long distance space travel and humanity spreads across the galaxy, there could in the future be quadrillions of people, growing at some kind of exponential rate. So accelerating the space race by even an hour is equivalent to bringing billions of new souls into existence.

Perhaps you're arguing as an illustration of the way this group of people think, in which case I understand your point.

It encodes a slight bias towards human existence being a positive thing for us humans, but I don't think it's the shakiest part of that reasoning.

They don't even do this.

If you're reasoning in purely logical and deductive way - it's blatantly obvious that living beings experience way more pain and suffering, than pleasure and joy. If you do the math, humanity getting wiped out in effect is the best thing that could happen.

Which is why accelerationism ignoring all the AGI risks is correct strategy presuming the AGI will either wipe us out (good outcome) or provide technologies that improve the human condition and reduce suffering (good outcome).

Logical and deductive reasoning based on completely baseless and obviously incorrect premises is flat out idiotic.

You can't deprive non-existent people out of anything.

And if you do, I hope you're ready for purely logical, deductive follow up - every droplet of sperm is sacred and should be used to impregnate.

You don't have to agree with any of this. I am not defending every idea the author has. But I recommend that book.

Of course it sounds ridiculous when you spell it out this way.

Of course the way your comment is written makes criticism sound silly.

In any case, EA smells strongly of “the ends justify the means” which most popular moral philosophies reject with strong arguments. One which resonates with me is that there are no “ends.” The path itself is the goal.

This is a false statement. Our entire modern world is built on the basis of the ends justify the means. Every time money is spent on long term infrastructure vs giving poor kids food right now, every time a war is fought, every time a doctor triages injuries at a disaster.

I'm sure exactly what you described was done plenty of times in ww1 and similar around that era, and seen as perfectly moral and rational.

Winning at things that align with your principle is a principle. If you don't care about principles, you don't care about what you're winning at, thereby making every victory hollow and meaningless. That is how you turn into a loser at everything you do.

How does this apply to actual charity? "Curing malaria is not the goal. Our experiences during voluntourism are the true goal."

It then moved into "you should work a soulless investment banking job so you can give more".

More recently it was "you should excise all expensive fun things from your life, and give 100% of your disposable income to a weird poly sex cult and/or their fraudulent paper hedge fund because they're smarter than you."

https://forum.effectivealtruism.org/posts/f6kg8T2Lp6rDqxWwG/...

Meanwhile, on an actual EA website: https://www.givewell.org/charities/top-charities

* Medicine to prevent malaria

* Nets to prevent malaria

* Supplements to prevent vitamin A deficiency

* Cash incentives for routine childhood vaccines

Its important to know which type of EA organization you are supporting before you donate, because the movement includes all three.

(And I assume that GiveWell top charities receive orders of magnitude more money, but I haven't actually checked the numbers.)

(1) The kind of Gatesian solutions they like to fund like mosquito nets are part of the problem, not part of the solution as I see it. If things are going to get better in Africa, it will be because Africans grow their economy and pay taxes and their governments can provide the services that they want. Expecting NGOs to do everything for them is the same kind of neoliberal thinking that has rotted state capacity in the core and set us up for a political crisis.

(2) It is one thing to do something wrong, realize it was a mistake, and then make amends. It's another thing to do plan to do something wrong and to try to offset it somehow. Many of the high paying jobs that EA wants young people to enter are "part of the problem" when it comes to declining stage capacity, legitimation crisis, and not dealing with immediate problems -- like the fact that one of these days there's going to be a heat wave that is a mass causality event.

Furthermore

(3) Time discounting is a central part of economic planning

https://en.wikipedia.org/wiki/Social_discount_rate

It is controversial as hell, but one of the many things the Soviet Union got wrong before the 1980s was planning with a discount rate of zero, which led to many economically and ecologically harmful projects. If you seriously think it should be zero you should also be considering whether anybody should work in the finance industry at all or if we should have dropped a hydrogen bomb on Exxon's headquarters yesterday. At some point speculations about the future are just speculation. When it comes to the nuclear waste issue, for instance, I don't think we have any idea what state people are going to be in 20,000 years. They might be really pissed that buried spent nuclear fuel some place they can't get at it. Even the plan to burn plutonium completely in fast breeder reactors has an air of unreality about it, even though it happens on a relatively short 1000 year timescale we can't be sure at all that anyone will be around to finish the job.

(4) If you are looking for low-probability events to worry about I think you could find a lot of them. If it was really a movement of free thinkers they'd be concerned about 4,000 horsemen of the apocalypse, not the 4 or so that they are allowed to talk about -- but talk about a bunch of people who'll cancel you if you "think different". Somehow climate change and legitimation crisis just get... ignored.

(5) Although it is run by people who say they are militant atheists, the movement has all the trappings of a religion, not least "The Singularity" was talked about by Jesuit Priest Teilhard de Chardin long before sci-fi writer Vernor Vinge used it as the hinge of a mystery novel.

The difficulty is in deriving any useful utility function from prices (even via preferences :), and as you know, econs can't rid themselves of that particular intrusive thought

https://mitsloan.mit.edu/sites/default/files/inline-files/So...

E: know any econs taking Habermas seriously ? Not a rhetorical q:

http://ecoport.org/storedReference/558800.pdf

[1] Though you might come to the conclusion that greeder people should have the money because they like it more

(Aside from the semi-tragic one to consider additive dilogarithms..)

One actionable (utility agnostic) suggestion: study the measureable consequences of (quantifiable) policy on carbon pricing, because this is already quite close to the uncontroversial bits

E: by "uncontroversial", I meant amongst the Orthodox econs, so not Graeber & sympathetic heterodox.

Nuclear waste issues are 99.9% present-day political/ideological. Huge portions of the Earth are uninhabitable due to climate and/or geology. Lead, mercury, arsenic, and other naturally-occurring poisons contaminate large areas. Volcanoes spew CO2 and toxic gasses by the megaton.

Vs. when is the last time you heard someone get excited over toxic waste left behind by the Roman Empire?

https://en.wikipedia.org/wiki/Mausoleum_of_Qin_Shi_Huang

Also, PaulHoule's original comment said "in 20,000 years". Cobalt 60 (for example) has a half-life of 5 1/4 years - so there really won't be any of it left by then.

No one is talking about stuffing cobalt 60 in yucca mountain (at least as far as I know).

And the tech to detect that you're digging into radioactive stuff is far simpler than the tech to detect that you're digging into some sort of chemical waste, or a failing old mine or tunnel.

If millennia-in-the-future humans care all that much about what we did with our nuclear waste, it'll either be political/ideological, or (as PaulHoule suggested) just one more "they didn't leave it somewhere really convenient for us" deal.

For archeologists, pretty much every time.

The problem with this argument is that the path to achieve this is unclear, and everyone who has tried has failed. In the absence of a clear path, it seems rational to set aside lofty ideals and do whatever good you can now.

> If you are looking for low-probability events to worry about I think you could find a lot of them.

Name one that hasn't already been considered that would be a serious threat to modern technological civilization.

> Although it is run by people who say they are militant atheists, the movement has all the trappings of a religion

Supposing you're correct this thought is incomplete. Take it to its logical conclusion: a religion centered around open rational debate is bad because...?

Similarly, Big Bang was talked about by Catholic priest Georges Lemaître, and Bayes' Theorem was invented by Presbyterian minister Thomas Bayes. Does that prove anything beyond the fact that there are many smart religious people?

This is the logic of someone who has failed to comprehend the core ideas of Calculus 101. You cannot use intuitive reasoning when it comes to infinite sums of numbers with extremely large uncertainties. All that results is making a fool out of yourself.

They use technical terms (eg expected value, KL divergence) in their verbal reasoning only to sound rational, but don’t ever mean to use those terms technically.

I quickly lost interest in Roko's Basilisk, but that is what brought me in the door and started me looking around the discussions. At first, it was quite seductive. There was a strange fearlessness there, a willingness to say and admit some things about humanity, our limitations and how we tend to think that other great thinkers maybe danced around in the past. After awhile it became clear that while there were a select few individuals who had found some balance between purely rational thinking and how reality actually works, most of the rest had their heads so far up their asses that they'd fart and call it a cool breeze. Reminded me of my brief obsession with Game Theory and realizing that even it's creators knew it's utility was not quite as advertised to the layman (as in it would not really help you predict or plan for anything at all, just model how decisions might be made).

Physics postgrads: "gauge"

Physics undergrads: "wavefunction"

Grade schoolers: "temperature"

These concepts definitely useful for the hw sets, no understanding needed (or expected!)

Most of criticisms are just "But they think they are better than us !" and the rest is "But sometimes they are wrong !"

I don't know about the community and couldn't care less but their writings have brought me some almost life saving fresh air in how to think about the world. It is very sad to me to read so many falsely elaborate responses from supposedly intelligent people having their ego hurt but in the end it reminds me why I like rationalists and I don't like most people.

Garak: Especially when they're neither.

Extra points for that comment's author implying that people who don't like the wrong and smug movement are unintelligent and protecting their egos, thus personally proving its smugness

As for smugness, it is subjective. Are those people smug ? Or are they talking passionately about some issue with the confidence of someone who feel what they are talking about and are expecting for it to resonate ? It's the eye of the beholder I guess.

For example what you call my smugness is what I would a slightly depressed attitude fueled by the fact that it's sometimes hard to relate to other people feelings and behavior.

Humans are generally better at perceiving threats than they are at putting those threats into words. When something seems "dangerous" abstractly, they will come up with words for why---but those words don't necessarily reflect the actual threat, because the threat might be hard to describe. Nevertheless the valence of their response reflects their actual emotion on the subject.

In this case: the rationalist philosophy basically creeps people out. There is something "insidious" about it. And this is not a delusion on the part of the people judging them: it really does threaten them, and likely for good reason. The explanation is something like "we extrapolate from the way that rationalists think and realize that their philosophy leads to dangerous conclusions." Some of these conclusions have already been made by the rationalists---like valuing people far away abstractly over people next door, by trying to quantify suffering and altruism like a math problem (or to place moral weight on animals over humans, or people in the future over people today). Other conclusions are just implied, waiting to be made later. But the human mind detects them anyway as implications of the way of thinking, and reacts accordingly: thinking like this is dangerous and should be argued against.

This extrapolation is hard to put into words, so everyone who tries to express their discomfort misses the target somewhat, and then, if you are the sort of person who only takes things literally, it sounds like they are all just attacking someone out of judgment or bitterness or something instead of for real reasons. But I can't emphasize this enough: their emotions are real, they're just failing to put them into words effectively. It's a skill issue. You will understand what's happening better if you understand that this is what's going on and then try to take their emotions seriously even if they are not communicating them very well.

So that's what's going on here. But I think I can also do a decent job of describing the actual problem that people have with the rationalist mindset. It's something like this:

Humans have an innate moral intuition that "personal" morality, the kind that takes care of themselves and their family and friends and community, is supposed to be sacrosanct: people are supposed to both practice it and protect the necessity of practicing it. We simply can't trust the world to be a safe place if people don't think of looking out for the people around them as a fundamental moral duty. And once those people are safe, protecting more people, such as a tribe or a nation or all of humanity or all of the planet, becomes permissible.

Sometimes people don't or can't practice this protection for various reasons, and that's morally fine, because it's a local problem that can be solved locally. But it's very insidious to turn around and justify not practicing it as a better way to live: "actually it's better not to behave morally; it's better to allocate resources to people far away; it's better to dedicate ourselves to fighting nebulous threats like AI safety or other X-risks instead of our neighbors; or, it's better to protect animals than people, because there are more of them". It's fine to work on important far-away problems once local problems are solved, if that's what you want. But it can't take priority, regardless of how the math works out. To work on global numbers-game problems instead of local problems, and to justify that with arguments, and to try to convince other people to also do that---that's dangerous as hell. It proves too much: it argues that humans at large ought to dismantle their personal moralities in favor of processing the world like a paperclip-maximizing robot. And that is exactly as dangerous as a paperclip-maximizing robot is. Just at a slower timescale.

(No surprise that this movement is popular among social outcasts, for whom local morality is going to feel less important, and (I suspect) autistic people, who probably experience less direct moral empathy for the people around them, as well as to the economically-insulated well-to-do tech-nerd types who are less likely to be directly exposed to suffering in their immediate communities.)

Ironically paperclip-maximizing-robots are exactly the thing that the rationalists are so worried about. They are a group of people who missed, and then disavowed, and now advocate disavowing, this "personal" morality, and unsurprisingly they view the world in a lens that doesn't include it, which means mostly being worried about problems of the same sort. But it provokes a strong negative reaction from everyone who thinks about the world in terms of that personal duty to safety, because that is the foundation of all morality, and is utterly essential to preserve, because it makes sure that whatever else you are doing doesn't go awry.

(edit: let me add that your aversion to the criticisms of rationalists is not unreasonable either. Given that you're parsing the criticisms as unreasonable, which they likely are (because of the skill issue), what you're seeing is a movement with value that seems to be being unfairly attacked. And you're right, the value is actually there! But the ultimate goal here is a synthesis: to get the value of the rationalist movement but to synthesize it with the recognition of the red flags that it sets off. Ignoring either side, the value or the critique, is ultimately counterproductive: the right goal is to synthesize both into a productive middle ground. (This is the arc of philosophy; it's what philosophy is. Not re-reading Plato.) The rationalists are probably morally correct in being motivated to highly-scaling actions e.g. the purview of "Effective Altruism". They are getting attacked for what they're discarding to do that, not for caring about it in the first place.)

There is something about a particular "narrowband" signaling approach, where a certain kind of purity is sought, with an expectation that, given enough explaining, you will finally get it, become enlightened, and convert to the ranks. A more "wideband" approach would at least admit observations like yours do exist and must be comprehensively addressed to the satisfaction of those who hold such beliefs vs to the satisfaction of those merely "stooping" to address them (again in the hopes they'll just see the light so everyone can get back to narrowband-ville).

edit: oh, also, I think that a good part of people's aversion to the rationalists is just a reaction to the narrowband quality itself, not to the content. People are well-aware of the sorts of things that narrowband self-justifying philosophies lead to, from countless examples, whether it's at the personal level (an unaccountable schoolteacher) or societal (a genocidal movement). We don't trust a group unless they specifically demonstrate non-narrowbandedness, which means being collectively willing to change their behavior in ways that don't make sense to them. Any movement that co-opts the idea of what is morally justifiable---who says that e.g. rationality is what produces truth and things that run counter to it do not---is inherently frightening.

Any group that focuses on their own goals of high paying jobs regardless of the morality of those jobs or how they contribute to the structural issues of society is not that good. Then donating money while otherwise being okay with the status quo —- not touching anything systemic in such an unjust world but supposedly focusing on morality is laughable.

In their defense they do try to do the calculations: is a high-paying job okay on net if you give most of the money away? depends on the job; depends on if somebody else would have it if you didn't; etc. Not that there is a rigorous way to do them but it's very much a group that does try.

I head not read any rationalist writing in a long time (and I didn't know about Scott's proximity), but the whole time I was reqding the article I was thinking the same thing you just wrote... "why are they afraid of AI, i.e. the ultimate rationalist taking over the world", maybe something deep inside of them has the same reaction to their own theories as you so eloquently put above.

The AI will do what its programmed to do, but its programmers morality may not match my own. What more scary is that it may be developed with the morality of a corporation rather than a person. (That is to say, no morals at all.)

I think its perfectly justifiable to be scared of a very powerful being with no morals stomping around!

Up to a certain level. Behind that level, the corp will be a tool of the AI. And even further that way, the corp will no longer be necessary.

[0]: https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...

Similar claims can be made about any structure of humans that exhibits gestalt intelligence, e.g. nations, stock markets, etc.

Maybe they want to do it in a way I’d consider just: By exercising their rights as individuals in their personal domains and effectively airing their arguments in the public sphere to win elections.

But my intuition is they think democracy and personal rights of the non-elect are part of the problem to rationalize around and over.

Would genuinely love to read some Rationalist discourse on this question.

Reading critiques of Hegel is a great starting point for this reading.

Whether you accept it or not though, there's lots of non-rationalist schools that reject the need for a "theory of power".

When Curtis Yarvin is at least in your orbit, these should not be surprising questions to get.

This sort of reasoning sounds great from 1000 feet up, but the longer you do it the closer you get to "I need to kill nearly all current humans to eliminate genetic diseases and control global warming and institute an absolute global rationalist dictatorship to prevent wars or humanity is doomed over the long run".

Or you get people who are working in a near panic to bring about godlike AI because they think that once the AI singularity happens the new AI God will look back in history and kill anybody who didn't work their hardest to bring it into existence because they assume an infinite mind will contain infinite cruelty.

I really like the depth of analysis in your comment, but I think there's one important element missing, which is that this is not an individual decision but a group heuristic to which individuals are then sensitized. Individuals don't typically go so far as to (consciously or unconsciously) extrapolate others' logic forward to decide that it's dangerous. Instead, people get creeped out when other people don't adhere to social patterns and principles that are normalized as safe in their culture, because the consequences are unknown and therefore potentially dangerous; or when they do adhere to patterns that are culturally believed to be dangerous. This can be used successfully to identify things that are really dangerous, but also has a high false positive rate (people with disabilities, gender identities, or physical characteristics that are not common or accepted within the beholder's culture can all trigger this, despite not posing any immediate/inherent threat) as well as a high false negative rate (many serial killers are noted to have been very charismatic, because they put effort into studying how to behave to not trigger this instinct). When we speak of something being normalized, we're talking about it becoming accepted by the mainstream so that it no longer triggers the ‘creepy’ response in the majority of individuals. As far as I can tell, the social conservative basically believes that the set of normalized things has been carefully evolved over many generations, and therefore should be maintained (or at least modified only very cautiously) even if we don't understand why they are as they are, while the social liberal believes that we the current generation are capable of making informed judgements about which things are and aren't harmful to a degree that we can (and therefore should) continuously iterate on that set to approach an ideal goal state in which it contains only things that are factually known to be harmful.

As an interesting aside, the ‘creepy’ emotion, (at least IIRC in women) is triggered not by obviously dangerous situations but by ambiguously dangerous situations, i.e. ones that don't obviously match the pattern of known safe or unsafe situations.

> Sometimes people don't or can't practice this protection for various reasons, and that's fine; it's a local problem that can be solved locally. But it's very insidious to turn around and justify not practicing it: "actually it's better not to behave morally; it's better to allocate resources to people far away; it's better to dedicate ourselves to fighting nebulous threats like AI safety or other X-risks instead of our neighbors".

The problem with the ‘us before them’ approach is that if two neighbourhoods each prioritize their local neighbourhood over the remote neighbourhood and compete (or go to war) to better their own neighbourhood at the cost of the other, generally both neighbourhoods are left worse off than they started, at least in the short term: both groups trying to make locally optimal choices leads (without further constraints) to globally highly suboptimal outcomes. In recognition of this a lot of people, not just capital-R Rationalists, now believe that at least in the abstract we should really be trying to optimize for global outcomes.

Whether anybody realistically has the computational ability to do so effectively is a different question, of course. Certainly I personally think the future-discounting ‘bias’ is a heuristic used to acknowledge the inherent uncertainty of any future outcome we might be trying to assign moral weight to, and should be accorded some respect. Perhaps you can make the same argument for the locality bias, but I guess that Rationalists (generally) either believe that you can't, or at least have a moral duty to optimize for the largest scope your computational power allows.

(...that said, progressivism has largely failed in dispelling this delusion. It is far too easy to feel as though progressivism/liberalism exists to prop up power hierarchies and economic disparities because in many ways it does, or has been co-opted to do that. I think on net it does not, but it should be much more cut-and-dry than it is. For that to be the case progressivism would need to find a way to effectively turn on its overlords/parasites, that is, rent-extracting capitalism and status-extracting moral elitism).

re: the first part of your reply. I sorta agree but I do think people do more extrapolation than you're saying on their own. The extrapolation is largely based on pattern-matching to known things: we have a rich literature (in the news, in art, in personal experience and storytelling) of failure modes of societies, which includes all kinds of examples of people inventing new moral rationalizations for things and using them to disregard personal morality. I think when people are extrapolating rationalists' ideas to find things that creep them out, they're largely pattern-matching to arguments they've seen in other places. But those arguments are, well, real arguments that require addressing.

And yeah, there are plenty of examples of people being afraid of things that today we think they should not have been afraid of. I tend to think that that's just how things go: it is the arc of social progress to figure out how to change things from unknown+frightening to known+benign. I won't fault anyone for being afraid of something they don't understand, but I will fault them for not being open-minded about it or being unempathetic or being cruel or not giving people chances to prove themselves.

All of this is rendered much more opaque and confusing by the fact that everyone places way too much stock in words, though. (e.g. the OP I was replying to who was taking all these criticisms of the rationalists at face-value). IMO this is a major trend that fucks royally with our ability as a society to make moral progress: we have come to believe that words supplant emotional intuition in a way that wrecks out ability to actually understand what people are upset about (I like to blame this trend for much of the modern political polarization). A small example of this is a case that I think everyone has experienced, which is a person discounting their own sense of creepiness from somebody else because they can't come up with a good reason to explain it and it feels unfair to treat someone coldly on a hunch. That should never have been possible: everyone should be trusting their hunches.

(which may seem to conflict with my preceding paragraph... should you trust your hunches or give people the chance to prove themselves? well, it's complicated, but it also really depends on what the result is. Avoiding someone personally because they creep you out is fine, but banning their way of life when it doesn't affect you at all is certainly not.)

If children around you are doing of an easily preventable disease, then yes, help them first! If they just need more arts programs, then you help the children dying in another country first.

But anyway this whole model follows from a basic set of beliefs about quantifying suffering and about what one's ethical responsibilities are, and it answers those in ways most people would find very bizarre by turning them into a math problem that assigns no special responsibility to the people around you. I think that is much more contentious and gross to most people than EA thinks it is. It can be hard to say exactly why in words, but that doesn't make it less true.

In college, I became a scale-dependent realist, which is to say, that I'm most confident of theories / knowledge in the 1-meter, 1-day, 1 m/s scales and increasingly skeptical of our understanding of things that are bigger/smaller, have longer/short timeframes, or faster velocities. Maybe there is a technical name for my position? But, it is mostly a skepticism about nearly unlimited extrapolation using brains that evolved under selection for reproduction at a certain scale. My position is not that we can't compute at different scales, but that we can't understand at other scales.

In practice, the rationalists appear to invert their confidence, with more confidence in quarks and light-years than daily experience.

Musing on the different failure-directions: Pretty much any terrible present thing against people can be rationalized by arguing that one gadzillion distant/future people are more important. That includes religious versions, where the stakes of the holy war may presented as all of future humanity being doomed to infinite torment. There are even some cults that pitch it retroactively: Offer to the priesthood to save all your ancestors who are in hell because of original sin.

The opposite would be to prioritize the near and immediate, culminating in a despotic god-king. This is somewhat more-familiar, we may have more cultural experience and moral tools for detection and prevention.

A check on either process would be that the denigrated real/nearby humans revolt. :p

This statement of yours makes no sense.

EAs by definition are attempting to remove the innate bias that discounts people far away by instead saying all lives are of equal worth.

>turning them into a math problem that assigns no special responsibility to the people around you

All lives are equal isn't a math problem. "Fuck it blow up the foreigners to keep oil prices low" is a math problem, it is a calculus that the US government has spent decades performing. (One that assigns zero value to lives outside the US.)

If $100 can save 1 life 10 blocks away from me or 5 lives in the next town over, what kind as asshole chooses to let 5 people die vs 1?

And since air travel is a thing, what the hell does "close to us" mean?

For that matter, from a purely selfish POV, helping lift other nations up to become fully advanced economies is hugely beneficial to me, and everyone on earth, in the long run. I'm damn thankful for all the aid my country gave to South Korea, the number of scientific advances that have come out of SK damn well paid for any tax dollars my grandparents paid on many orders of magnitude times over.

> It can be hard to say exactly why in words, but that doesn't make it less true.

This is the part where I shout racism.

Because history has shown it isn't about people being far or close in distance, but rather in how those people look.

Americans have shot down multiple social benefit programs because, and these are what people who voted against those programs directly said was their reasons "white people don't want black people getting the same help white people get."

Whites in America have voted, repeatedly, to keep themselves poor rather than lift themselves and black families out of poverty at the same time.

Of course Americans think helping people in Africa is "weird".

The thing about strict-utilitarian-morality is that it can't comprehend any other kind of morality, because it evaluates the morality of... moralities... on its own utilitarian basis. And then of course it wins over the others: it's evaluating them using itself!

There are entirely different ethical systems that are not utilitarian which (it seems) most people hold and innately use (the "personal morality" I'm talking about in my earlier post). They are hard to comprehend rationally, but that doesn't make them less real. Strict-utilitarianism seems "correct" in a way that personal morality does not because you are working from a premise "only things that I can understand like math problems can be true". But what I observe in the world is that people's fear of the rationalist/EA mindset comes from the fact that they empirically find this way of thinking to be insidious. Their morality specifically disagrees with that way of thinking: it is not the case that truth comes from scrutable math problems; that is not the point of moral action to them.

The EA philosophy may be put as "well sure but you could change to the math-problem version, it's better". But what I observe is that people largely don't want to. There is a purpose to their choice of moral framework; it's not that they're looking at them all in a vacuum and picking the most mathematically sound one. They have an intrinsic need to keep the people around them safe and they're picking the one that does that best. EA on the other hand is great if everyone around you is safe and you have lots of extra spending money and what you're maximizing for is the feeling of being a good person. But it is not the only way to conceive of moral action, and if you think it is, you're too inside of it to see out.

I'll reiterate I am trying to describe what I see happening when people resist and protest rationalism (and why their complaints "miss" slightly---because IMO they don't have the language to talk about this stuff but they are still afraid of it). I'm sympathetic to EA largely, but I think it misses important things that are crippling it, of the variety above: an inability to recognize other people's moralities and needs and fears doesn't make them go away; it just makes them hate you.

I can comprehend them just fine, but I have a deep seated objection to any system of morality that leaves behind giant piles of dead bodies. We should be trying to minimize the size of the pile of dead bodies (and ideally eliminate the pile altogether!)

Any system or morality that boils down to "I don't care about that pile of dead bodies being huge because those people look different" is in fact not a system morality at all.

The job of a system of morality is to synthesize all the things we want to happen / want to prevent happening into a way of making decisions. One such thing is piles of dead bodies. Another is one's natural moral instincts, like their need to take care of their family, or the feeling of responsibility to invest time and energy into improving their future or their community or repairing justice or helping people who need help, or to attend to their needs for art and meaning and fun and love and respect. A coherent moral system synthesizes these all and figures out how much priority to allocate to each thing in a way that is reasonable and productive.

Any system of morality that takes one of these criteria and discards the rest of them is not a system of morality at all, in the very literal sense that nobody will do it. Most people won't sell out one of their moral impulses for the others, and EA/rationalism feels like it asks them too, since it asks them to place zero value in a lot of things that they inherently place moral value in, and so they find it creepy and weird. (It doesn't ask that explicitly; it asks it by omission. By never considering any other morality and being incapable of considering them, because they are not easily quantifiable/made logical, it asks you to accept a framework that sets you up to ignore most of your needs.)

My angle here is that I'm trying to describe what I believe is already happening. I'm not advocating it; it's already there, like a law of physics.

I think another part of it is a sort of healthy nativism or in-group preference or whatever you want to call it. It rubs people the wrong way when you say that you care about someone in a different country as much as you care about your neighbors. That’s just…antisocial. Taken to its logical conclusion, a “rationalist” should not only donate all of their disposable income to global charities, they should also find a way to steal as much as possible from their neighbors and donate that, too. After all, those. Holden in Africa need the money much more than their pampered western neighbors.

Not only that, but this is exactly the kind of scenario where we should be giving those signals the most weight: The individual estimating whether to join up with a tribe. (As opposed to, say, bad feelings about doing calculus.)

Not only does it involve humans-predicting-humans (where we have a rather privileged set of tools) but there have been millions of years of selective pressure to be decent at it.

People benefit from a sense of a family, a sense of community. It helps us feel more secure, both personally and to our loved ones.

I think the more I view things through this lenses the downstream benefits I see.

One thing I'd like to add though is that I do think there is an additional piece being discarded irrationally. They tend to highly undervalue everything you're describing. Humans aren't Vulcans. By being so obsessed with the risks of paperclip-maximizing-robots they devalue the risks of humans being the irrational animals they are.

This is why many on the left criticize them for being right wing. Not because they are, well some might be, but because they are incredibly easy to distract from what is being communicated by focusing too much on what is being said. That might be a bad phrasing but what I mean is that when you look at this piece from last year about prison sentence length and crime rates by Scott Alexander[0] nothing he says is genuinely unreasonable. He's generally evaluating the data fairly and rationally. Some might disagree there but that's not my point. My point is that he's talking to a nonexistant group. The right largely believes that punishment is the point of prison. They might _say_ the goal is to reduce crime, but they are communicating based on a set of beliefs that strongly favors punitive measures for their own sake. This causes a piece like that to miss the forest through the trees and can be seen by those on the left as functionally right wing propaganda.

Most people are not rational. Maybe some day they will be but until then it is dangerous to assume and act as if they are. This makes me see the rationalists as actually rather irrational.

0: https://www.astralcodexten.com/p/prison-and-crime-much-more-...

Part of the reason I enjoy rationalist discourse more is because, even if they are unabashedly utilitarian, they try to rigorously derive philosophy. Most internet discourse on philosophy is, as you say, just vaguely derived around gut feelings. But philosophy can and has been thought of rigorously. Virtue ethics and continental morality are both schools of thought that reject utilitarian ethics but are much more meaty than the sort of internet "no but my neighbors" intuition that you see in full force, and the weird insistence that these internet commenters continue to use their vague moral intuition without being rigorous about their own thoughts.

I never quite realized the connection between this creepy feeling and the Bleak House criticism of far-flung philanthropy, until this comment.

Being able to do that is pretty much "entry level cognition" for a lot of us. You should be doing that yourself and doing it all the time if you want to play with the big kids.

One of the things I really miss about the old nerds-only programmer's pit setup was the amount of room we had for instruction, especially regarding social issues. The scenes from the college department in Wargames were really on the nose, but highlight a form of education that was unavoidable if you couldn't just dip out of a conversation.

Accused of being unable to reexamine your base principles, you respond:

> rationalists, who are possibly the only meaningful social group in existence to celebrate changing their minds in response to new evidence

Which is exactly the kind of base principle that could use some reexamination.

The primary issues as others have noted is they focus on people going to the highest paying jobs without much care for morality of the jobs. Ergo they are fine being net negatives in terms of their work and philosophy.

All they do is donate money. Donations don’t fix society. Nothing changes structurally. No root problems are looked at.

They largely ignore capitalism’s faults or when I’ve seen them talk about, it’s done in a way of superficially decrying capitalist issues but then largely going along with them. Which ties into how they focus on high paying jobs regardless of morality (I’m exaggerating here but the overall point is correct).

—

HN is not intelligent when it comes to politics or the world. The avg person here is a western chauvinist with little political knowledge but a defensive ego about it. No need to be sad about this comment page.

I don't think any "Rationalists" I ever met would actually consider concepts like scientific method...

In that case I don't think you've met any of the people under discussion.

Do you have examples of that? I have a different perception, most of the EAs I've met are very grounded and sharp.

For example the most recent issue of their newsletter: https://us8.campaign-archive.com/?e=7023019c13&u=52b028e7f79...

I'm not sure where there are any “hypothetical logical thought exercises” that “end up coming to insane conclusions” in there.

For the first part where you say “not realizing that their initial conditions were highly artificial so any conclusion they reach is only of academic value” this is quite the opposite of my experience with them. They are very receptive to criticism and reconsider their point of view in reaction to that.

They are generally well-aware of the limits of data-driven initiatives and the dangers of indulging into purely abstract thinking that can lead to conclusions that indeed don't make sense.

The newsletter is of course far more to the point than that, but even then you'll notice half of it is devoted to understanding the emotional state and intentions of LLMs...

It is of course entirely possible to identify as an "Effective Altruist" whilst making larger-than-averaged donations to charities with rigorous efficacy metrics and otherwise being completely normal, but that's not the centre of EA debate....

EAs gave $1,886,513,058 through GiveWell[1], and there is 0 AI stuff in there (you can search in the linked Airtable spreadsheet).

There is also a whole movement for doing a lifetime commitment to give 10% of your earnings to charity. 9,880 people took the pledge so far[2].

[1] https://airtable.com/appGuFtOIb1eodoBu/shr1EzngorAlEzziP/tbl...

[2] https://www.givingwhatwecan.org/pledge

GiveWell continues to plow its own rigorous international development-focused furrow, but it's cofounder, once noted for calling out everyone else for lack of rigour in their evidence base, has moved on to fluffy essays about how this is probably "the most important century" because it's either AI armageddon or maybe his wife's $61B startup will save us all...

I also believe that idealistic people will go to great lengths to convince themselves that their desired outcome is, in fact, the moral one. It starts by saying things like, "Well, what is harm, actually..." and then constructing a definition that supports the conclusions they've already arrived at.

I'm quite sure Sam Bankman-Fried did not believe he was harming anybody when he lost/stole/defrauded his investors and depositors' money.

What is your alternative? What's your framework that makes you contribute to malaria prevention more or more effectively than EAs do? Or is the claim instead that people should shut down conversation within EA that strays from the EA mode?

I am plenty happy to simp for the Gates foundation, but I think it's important to acknowledge that becoming Bill Gates to support charity is not a strategy the average person can replicate. The question for me is how do I live my life to support the causes I care about, not who lives more impressive lives than me.

How much more do they need to give before you will change your mind about whether “EA's actually want to do something about malaria”?

[1] https://www.givewell.org/all-grants-fund

[2] https://airtable.com/appGuFtOIb1eodoBu/shr1EzngorAlEzziP/tbl...

If you exclude "nations" then it does look to be the Church: "The Church operates more than 140,000 schools, 10,000 orphanages, 5,000 hospitals and some 16,000 other health clinics". Caritas, the relevant charitable umbrella organization, gives $2-4b per year on its own, and that's not including the many, many operations run by religious orders not under that umbrella, or by the hundreds of thousands of parishes around the world (most of which operate charitable operations of their own).

And yet, rationalists are totally happy criticizing the Catholic Church -- not that I'm complaining, but it seems a bit hypocritical.

Similarly, it's not like government funding is an overlooked part of EA. Working on government and government aid programs is something EA talks about, high leverage areas like policy especially. If there's a more standard government role that an individual can take that has better outcomes than what EAs do, that would be an important update and I'd be interested in hearing it. But the criticism that EA is just not large enough is hard to action on, and more of a work in progress than a moral failing.

https://www.greaterwrong.com/posts/5JB4dn7hEvZCYDvCe/a-very-...

https://www.greaterwrong.com/posts/MHBJ436AiKyN83DCB/thinkin...

https://www.greaterwrong.com/posts/58ajesm2C38wi3WSJ/insect-...

There has to be (or ought to be) a name for this kind of epistemological fallacy, where in pursuit of truth, the pursuit of logical sophistication and soundness between starting assumptions (or first principles) and conclusions becomes functionally way more important than carefully evaluating and thoughtfully choosing the right starting assumptions (and being willing to change them when they are found to be inconsistent with sound observation and interpretation).

“[...] Clevinger was one of those people with lots of intelligence and no brains, and everyone knew it except those who soon found it out. In short, he was a dope." - Joseph Heller, Catch-22 https://www.goodreads.com/quotes/7522733-in-short-clevinger-...

HN should be better than this.

Can people suffer from that impairment? Is that possible? If not, please explain how wrong assumptions can be eliminated without actively looking for them. If the impairment is real, what would you call its victims? Pick your own terminology.

Calling someone a dumbass in this situation is a kindness, because the assumption is that they're capable of not being one with a little self-reflection.

First they laugh...

I can't see how insulting someone pursues their well-being in any way.

Maybe it's similar to the "Good Student" picture. Bright within a given assignment, but taking the assignment to be immutable, or taking no interest in where the assignment comes from.

Or I've heard a saying "nobody will pay you to solve problems that they've already defined clearly."

Isn't there a lot of overlap between the two groups?

I recently read a great book that examines these various groups and their commonality: More Everything Forever: AI Overlords, Space Empires, and Silicon Valley's Crusade to Control the Fate of Humanity by Adam Becker. Highly recommended.

Logic requires properties of metaphysical objectivity.

If you use the true meaning of words it would be called irrationality, delusion, sophism, or fallacy when such things are claimed true when in fact they are false.

I also think the ambiguity of meaning in natural language is why statistical llms are so popular with this crowd. You don't need to think about meaning and parsing. Whatever the llm assumes is the meaning is whatever the meaning is.

Ironically, that reminds me of "37 Ways That Words Can Be Wrong", written by Eliezer Yudkowsky in 2008...

I have read Effective Altruists like that. But I also remember seeing a lot of money donated to a bunch of really decent sounding causes because someone spent 5 minutes asking themselves what they wanted their donation to maximise, decided on "Lives saved" and figured out who is doing the best at that.

For anyone else reading, a good example of what EAs do can be seen with the GiveWell charity: https://www.givewell.org/top-charities-fund

Lots of anti-malaria and vitamin stuff (as a cheap way to save lots of lives). There are also tons of EA animal charities too, such as Humane League: https://thehumaneleague.org/our-impact

If they want to donate to charity, they can just donate. You don't gotta make a religion out of it.

Honestly thought they were the same people

https://www.lesswrong.com/ and https://forum.effectivealtruism.org/ run on the same software, and some people have an account at both of them, but they are generally separate web communities.

"I haven't done anything!" - A Serious Man

Yes, rationalism is not a substitute for humility or fallibility. However, rationalism is an important counterpoint to humanity, which is orthogonal to rationalism. But really, being rational is only binary - you cant be anything other than rational or irrational. You're either doing what's best or you're not. That's just a hard pill for most people to swallow.

To use the popular metaphor, people are drowning all over the world and we're all choosing not to save them because we don't want to ruin our shoes. Look in the mirror and try and comprehend how selfish we are.

>I guess I'm a rationalist now.

>Aren't you the guy who's always getting into arguments who's always right?

[1] S.A. actually quoted the person as follows: "You’re Scott Aaronson?! The quantum physicist who’s always getting into arguments on the Internet, and who’s essentially always right, but who sustains an unreasonable amount of psychic damage in the process?" which differs in several ways from what reverendsteveii falsely presents as a direct quotation.

Substitute God with AI or the concept of rationality and use "first principles"/Bayesianism in an extremely dogmatic manner similar to Catechism and you have the Rationalist/AI Alignment/Effective Altruist movement.

Ironically, this is how plenty of religious movements started off - basically as formalizations of philosophy and ethics that fused with what is basically lore and worldbuilding.

Whenever I try to get an answer of HOW (as in the attack path), I keep getting a deus ex machina. Reverting to a deus ex machina in a self purported Rationalist movement is inherently irrational. And that's where I feel the crux of the issue is - it's called a "Rationalist" movement, but rationalism (as in the process of synthesizing information using a heuristic) is secondary to the overarching theme of techno-millenarianism.

This is why I feel rationalism is for all intents and purposes a "secular religion" - it's used by people to scratch an itch that religion often was used as well, and the same Judeo-Christian tropes are basically adopted in an obfuscated manner. Unsurprisingly, Eliezer Yudkowsky is an ex-talmid.

There's nothing wrong with that, but hiding behind the guise of being "rational" is dumb when the core belief is inherently irrational.

I take it this is what you have in mind when you say that whenever you ask for an "attack path" you keep getting a deus ex machina. But it seems to me like a pretty weak basis for calling Yudkowsky's position on this a religion.

(Not all people who consider themselves rationalists agree with Yudkowsky about how big a risk prospective superintelligent AI is. Are you taking "the Rationalist movement" to mean only the ones who agree with Yudkowsky about that?)

> Unsurprisingly, Eliezer Yudkowsky is an ex-talmid

So far as I can tell this is completely untrue unless it just means "Yudkowsky is from a Jewish family". (I hope you would not endorse taking "X is from a Jewish family" as good evidence that X is irrationally prone to religious thinking.)

Agree to disagree.

> So far as I can tell this is completely untrue

I was under the impression EY attended Yeshivat Sha'alvim (the USC of Yeshivas - rigorous and well regarded, but a "warmer" student body), but that was his brother. That said, EY is absolutely from a Daatim or Chabad household given that his brother attended Yeshivat Sha'alvim - and they are not mainstream in the Orthodox Jewish community.

And the feel and zeitgeist around the rationalist community with it's veneration of a couple core people like EY or Scott Alexander does feel similar to the veneration a subset of people would do for Baba Sali or Alter Rebbe in those communities.

Let's take the chess analogy. I take it you agree that I would very reliably lose if I played Magnus Carlsen at chess; he's got more than 1000 Elo points on me. But I couldn't tell you the "attack path" he would use. I mean, I could say vague things like "probably he will spot tactical errors I make and win material, and in the unlikely event that I don't make any he will just make better moves than me and gradually improve his position until mine collapses", but that's the equivalent of things like "the AI will get some of the things it wants by being superhumanly persuasive" or "the AI will be able to figure out scientific/engineering things much better than us and that will give it an advantage" which Yudkowsky can also say. I won't be able to tell you in advance what mistakes I will make or where my pawn structure will be weak or whatever.

Does this mean that, for you, if I cared enough about my inevitable defeat at Carlsen's hands that expectation would be religious?

To me it seems obvious that it wouldn't, and that if Yudkowsky's (or other rationalists') position on AI is religious then it can't be just because one important argument they make has a step in it where they can't fill out all the details. I am pretty sure you have other things in mind too that you haven't made so explicit.

(The specific other things I've heard people cite as reasons why rationalism is really a religion also, individually and collectively, seem very unconvincing to me. But if you throw 'em all in then we are in what seems to me like more reasonable agree-to-disagree territory.)

> That said, EY is absolutely from a Daatim or Chabad household

I think holding that against him, as you seem to be doing, is contemptible. If his ideas are wrong, they're fair game, but insinuating that we should be suspicious of his ideas because of the religion of his family, which he has rejected? Please, no. That goes nowhere good.

The crazies and blind among humanity today can't think like that, its a deficiency people have, but they are still dependent on a group of people that are capable of that. A group that they are intent on ostracizing and depriving existence from in various forms.

You seem so wound up in the circular Paulo Freire based perspective that you can't think or see.

Bring things back to reality. If someone punches you in the face, you feel that fist hitting your face. You know someone punched you in the face. Its objective.

Imagine for a second and just assume that these people are right in their warnings, that everything they see is what you see, and all you can see is when you tip over a particular domino that has been tipped over in the past, a chain of dominoes falls over and at the end is the end of organized civilized society which tips over the ability to produce food.

For the purpose of this thought experiment, the end of the world is visible and almost here, and you can't change those dominoes after they've tipped, and worse you see the majority of people trying to tip those dominoes over for short term profit believing nothing they ever do can break everything.

Would you not be frothing at the mouth trying to get everyone you cared about to a point where they pry that domino up before it falls? so you and your children will survive? It is something you can't unsee, it is a thing that cannot be undone. Its coming. What do you do? If you are sane, you try with everything you have to help them keep it from toppling.

Now peal this thought back a moment, adjust it where it is still true, but you can't see it and you can only believe what you see.

Would you approach this differently given knowledge of the full consequence knowing that some people can see more than you? Would you walk out onto a seemingly visibly stable bridge that an engineer has said not to walk out on? Would you put yourself in front of a dam cracks running up the side, when an evacuation order was given? What would the consequence be for doing that if you led along your family and children to such places ignoring these things?

There are quite a lot of indirect principles that used to be taught which are no longer taught to the average person and this blinds them because they do not recognize it and recognition is the first thing you need to be able to act and adapt.

People who cannot adapt fail Darwin's fitness. Given all potential outcomes in the grand scheme of things, as complexity increases 99% of all outcomes are death vs life at 1%.

It is only through great care that we carry things forward to the future, and empower our children to be able to adapt to the environments we create.

Finally, we have knowledge of non-linear chaotic systems where adaptability fails because of hysteresis, where no matter how much one prepares the majority given sufficient size will die, and worse there are cohorts of people who are ensuring the environment we will soon live in is this type of environment.

Do you know how to build an organized society from scratch? If there is no reasonable plan, then you are planning to fail. Rather than make it worse through inaction, get out of the way so someone can make it better.

Have you ever read Scott Alexander's blog (Slate Star Codex, now Astral Codex X)? It's full of doubt and self-questioning. The guy even keeps a public list of his mistakes:

https://www.astralcodexten.com/p/mistakes

I'll admit my only touchpoint to the "rationalist community" is this blog, but I sure don't get "full of themselves" from that. Quite the contrary.

Perhaps on a meta level. If you already have high confidence in something, reasoning it out again may be a waste of time. But of course the rational answer to a problem comes from reasoning about it; and of course chains of reasoning can be traced back to first principles.

> And the general absolutist tone of the community. The people involved all seem very... Full of themselves ? They don't really ever show a sense of "hey, I've got a thought, maybe I haven't considered all angles to it, maybe I'm wrong - but here it is". The type of people that would be embarrassed to not have an opinion on a topic or say "I don't know"

Doing rationalism properly is hard, which is the main reason that the concept "rationalism" exists and is invoked in the first place.