Is it a cult? It's at least cult-ish. It quacks. Maybe not a duck but it's waterfowl.
All I'm really saying is people should look more deeply into this stuff. If they do, I think they'll see what I'm talking about. It is not hard to find.
They don't what to know so they consciously or unconsciously shut their eyes.
At least Anthropic is considering safety and forcing the conversation. They are saying "no" to helping develop AI-weapons for the Pentagon. Maybe the doomsday stuff is a marketing scheme, but I don't care because I think that it's better to move slowly and carefully and to regulate the industry. We have learned a lot since the prior "cult" for which the rally cry was "move fast and break things."
If they truly believed this stuff had a meaningful chance of being an X-risk to humans, they should stop building it and stop pursuing a giant IPO to finance building it.
Counterpoint: the many long papers and posts discussing why this is a flawed strategy.If you’re going to write multiple paragraphs deriding scientists, you should at least bother to engage with the basics of their findings. Not just spin up bullshit about BDSM orgies and call it a day.
Your reply hinges on "every work", which is far from necessary here. You could read a couple blog posts, a wikipedia page or two, or, yes, some books and papers.
In other words: do you have to read every climate paper to vote for candidates interested in tackling climate change?
If starving the project of money and resources by immediately stopping work on it and refusing to do more work on it won't work, then regulation sure as fuck won't do anything.
The sort of people who will go ahead full steam on -say- a biosphere annihilator that they've demonstrated that they are incapable of building safely (and admitted their knowledge of their own incapability) are not the sort of people who will closely follow laws and regulations that constrain their activities in ways they disagree with.
As we've seen here recently, when you have trillions of dollars at your disposal, you can buy a lot of "turning a blind eye" to your past, current, and future violations of relevant laws and regulations. And as we saw in the US in 2008, if you get the right people scared enough, you can get your past violations of law made retroactively legal.
If starving the project of money and resources by immediately stopping work on it and refusing to do more work on it won't work, then regulation sure as fuck won't do anything.
...again, you're just musing about weak strawmen. The arguments tie into the incentives and tendencies inherent to Capitalism and Nationalism. Thinking up unrelated hypotheticals is not a counterpoint.I'm increasingly believing that, yes, you can use AI to build dangerous systems, but the most dangerous things is that narrative of inevitability.
Humans actually have a lot of agency in this, and it will take humans to decide to do something really dangerous with AI. The inevitability narrative conveniently absolves them of any responsibility.
I think most people in power know that, but they also know that we're in for a heavy hitting recession if this stops - and no one wants to be the one "causing" this by speaking up against it too much.
Which [put on your tinfoil hats now] might also be why the whole jensen story is how it is, because none of it is "active action". It surely makes people angry, but it doesn't make them angry in a way where they show up at someone's door and go "your decisions are responsible for my economic hardship".
It is probably not that and just random chance, but, functionally, it might be a way of letting this crash that doesn't end up with this style of vigilante/lunatic going around.
One was the way writers would assert premises as if they were axiomatic when they were not and proceed to reason from them to a conclusion without checking the result empirically. This is more like medieval Scholasticism than science. Of course heavier objects fall faster than lighter objects! Why bother doing the experiment? Isn't it obvious?
https://en.wikipedia.org/wiki/Scholasticism
I think Rationalism is sort of a neo-Scholasticism, not in the specific sense of trying to reconcile ancient thought with Catholicism (though there's a weird neo-Catholic strain in there somewhere) but in the sense of believing in reasoning from asserted truth to correct outcome. "I am smart and my assertions feel right so I'm correct."
Another was a constant odor of biological determinism, crypto-racism, and IQ elitism. I don't even consider myself a "woke leftist" per se. I dislike these things because I legitimately think they're either wrong or irrelevant as the case may be.
I don't think it's a coincidence that neoreaction and similar things emerged from these circles. (Yes I'm aware that many today disavow those things.) The soil was fertile.
Same with Longtermism which is just a rebrand of utilitarianism. I can make a solid Rational(tm) argument that killing everyone with a disability or a measured IQ below, say, 80, would improve long term utilitarian outcomes for the human race and would have a high probability of reducing suffering in the long term. I'm absolutely certain I could write a convincing Rationalist-style essay on this. Ready to push the button? There's a reason this kind of pure utilitarian thought has a poor reputation in actual ethics circles.
EA is full of creeps like Sam Bankman Fried, scams, and people who seem to use it to surround their ordinary profit motives with a cloud of smug.
TL;DR: Know the quality of a tree by its fruit, and I am not impressed by the fruit.
I don’t rank “doom” very high though. Bad things? Sure. But good things too. Big lever.
The huge issue I have with the safety crowd, especially this one, is that one of the scenarios I fear the most and also think is realistic is monopolization. I’m afraid of what happens if small numbers of people and governments completely monopolize powerful AI and then use it as a force multiplier to become kings of the world more or less.
My kids living under enhanced AI augmented absolute god kings is IMO a “doom” scenario almost on par with mass death. You could argue it’s a future with more net suffering than one where we are just snuffed out. It could even lead to that, just with humans pushing the button.
This isn’t as far fetched as the other scenarios because we already know for a fact that humans behave this way. We have tons of historical examples. Even if they start benevolent, absolute power distorts the human mind and tends to make people lose touch with reality.
The safety measures the safetyists advocate — banning open models, regulating training, forcing research through gates they control — are precisely the measures most likely to lead to this outcome.
(Not exactly ideal under Trump, so let's say it happens in 2029 under a Democratic party administration.)
I heard someone on a group I'm in humorously say it's the first IPO of an apocalyptic doomsday cult.
The seeming indoctrination of the incoming Democrats from the most recent electoral cycle is really troubling. I've wondered if maybe a good way to fight it is to point out that this cult has some problematic views on race/IQ, biological determinism, and has a lot of figures in it with really troubling histories around sex and consent. Search for "Rationalist sex cult" and you'll find endless articles about "Rationalist houses," BDSM orgies that play loose with consent, employees feeling like they're being pressured to participate in certain extracurricular activities, etc.
I feel like these allegations pouring out of other circles would be a five alarm fire, but for some reason the Rationalist orbit around Anthropic and OpenAI and the whole San Francisco Rationalist cult scene gets a pass.