One was the way writers would assert premises as if they were axiomatic when they were not and proceed to reason from them to a conclusion without checking the result empirically. This is more like medieval Scholasticism than science. Of course heavier objects fall faster than lighter objects! Why bother doing the experiment? Isn't it obvious?
https://en.wikipedia.org/wiki/Scholasticism
I think Rationalism is sort of a neo-Scholasticism, not in the specific sense of trying to reconcile ancient thought with Catholicism (though there's a weird neo-Catholic strain in there somewhere) but in the sense of believing in reasoning from asserted truth to correct outcome. "I am smart and my assertions feel right so I'm correct."
Another was a constant odor of biological determinism, crypto-racism, and IQ elitism. I don't even consider myself a "woke leftist" per se. I dislike these things because I legitimately think they're either wrong or irrelevant as the case may be.
I don't think it's a coincidence that neoreaction and similar things emerged from these circles. (Yes I'm aware that many today disavow those things.) The soil was fertile.
Same with Longtermism which is just a rebrand of utilitarianism. I can make a solid Rational(tm) argument that killing everyone with a disability or a measured IQ below, say, 80, would improve long term utilitarian outcomes for the human race and would have a high probability of reducing suffering in the long term. I'm absolutely certain I could write a convincing Rationalist-style essay on this. Ready to push the button? There's a reason this kind of pure utilitarian thought has a poor reputation in actual ethics circles.
EA is full of creeps like Sam Bankman Fried, scams, and people who seem to use it to surround their ordinary profit motives with a cloud of smug.
TL;DR: Know the quality of a tree by its fruit, and I am not impressed by the fruit.
I don’t rank “doom” very high though. Bad things? Sure. But good things too. Big lever.
The huge issue I have with the safety crowd, especially this one, is that one of the scenarios I fear the most and also think is realistic is monopolization. I’m afraid of what happens if small numbers of people and governments completely monopolize powerful AI and then use it as a force multiplier to become kings of the world more or less.
My kids living under enhanced AI augmented absolute god kings is IMO a “doom” scenario almost on par with mass death. You could argue it’s a future with more net suffering than one where we are just snuffed out. It could even lead to that, just with humans pushing the button.
This isn’t as far fetched as the other scenarios because we already know for a fact that humans behave this way. We have tons of historical examples. Even if they start benevolent, absolute power distorts the human mind and tends to make people lose touch with reality.
The safety measures the safetyists advocate — banning open models, regulating training, forcing research through gates they control — are precisely the measures most likely to lead to this outcome.
(Not exactly ideal under Trump, so let's say it happens in 2029 under a Democratic party administration.)