News for hackers. Stuff that matters.
Re:OpenAI safety leader quits, warning AI company's culture ... by danpalmer on Saturday October 03, 2026 @07:06PM (#49948661)
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?

A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Saturday October 03, 2026 @07:32PM (#49948849)
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Saturday October 03, 2026 @08:12PM (#49949176)
What makes you think, the scenarios in question here would be "silly"?

Is it that "chatbots" can't come out of the screen to immediately harm you physically?

Let's say they simply manage to take down the internet. How many would die?

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Saturday October 03, 2026 @08:33PM (#49949322)
So what. Various attackers managed to take down large chunks of the Internet on a frequent basis before LLMs even existed. This killed very few people. The great thing about the Internet is how resilient it is.
Re:OpenAI safety leader quits, warning AI company's culture ... by bravetraveler on Saturday October 03, 2026 @08:45PM (#49949396)
Darling companies of this very website have mistakenly brought down large portions of the internet thanks to our old friend BGP. No attacks required, just small oversights and unfortunate concentration on the business/IP space! This happens regularly.

Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.

Re:OpenAI safety leader quits, warning AI company's culture ... by goolz on Saturday October 03, 2026 @08:35PM (#49949328)
It is that they are chatbots. If it were real AI, an actual singularity, I would worry, maybe. But it isn’t. They are absurdly powerful automation tools that can handle logic better than a human can dream of. They take care of the grunt minutiae without complaint. But they are not going to end the world in their current form.
Re:OpenAI safety leader quits, warning AI company's culture ... by pixl97 on Saturday October 03, 2026 @09:21PM (#49949656)
So we're going to wait till after they can adopt a form they can end the world in?

And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.

And that's not even counting 'minor' problems like society falling apart.

Re:OpenAI safety leader quits, warning AI company's culture ... by SV_BubbleTime on Saturday October 03, 2026 @09:00PM (#49949511)
> Let's say they simply manage to take down the internet.

geez, don’t threaten me with a good time.

I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.

Re:OpenAI safety leader quits, warning AI company's culture ... by BLKNSLVR on Saturday October 03, 2026 @09:23PM (#49949665)
That Simpsons episode when Marge managed to get Itchy and Scratchy banned briefly.

The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @08:24PM (#49949265)
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.

Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".

Re:OpenAI safety leader quits, warning AI company's culture ... by slashdave on Saturday October 03, 2026 @10:10PM (#49949918)
Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.

A pandemic is perfectly plausible.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @10:42PM (#49950095)
Everything is plausible with the benefit of hindsight. E.g. "Tintin on the Moon" predated the Moon landings. From the perspective of e.g. 1850, the idea of landing on the Moon was "extraordinary and without a basis in known science or technology".
Re:OpenAI safety leader quits, warning AI company's culture ... by biophysboy on Saturday October 03, 2026 @09:47PM (#49949790)
There’s nothing wrong with bold predictions, but they should be paired with good methods. The doom predictions are not paired with good explanation.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @10:01PM (#49949876)
Doomers have invested a ton of time in explanations. Here are a couple just off the top of my head:

https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...

https://www.youtube.com/watch?v=7wy3xyoXYt8

Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1

Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Saturday October 03, 2026 @10:13PM (#49949949)
Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!
Re:OpenAI safety leader quits, warning AI company's culture ... by throwaway27448 on Saturday October 03, 2026 @11:39PM (#49950359)
> If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it.

"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.

Re:OpenAI safety leader quits, warning AI company's culture ... by anon7725 on Saturday October 03, 2026 @08:59PM (#49949507)
Yeah except pandemics are not novel, unlike AI doom scenarios.
Re:OpenAI safety leader quits, warning AI company's culture ... by estearum on Saturday October 03, 2026 @09:08PM (#49949566)
It's good that new bad things never happen.
Re:OpenAI safety leader quits, warning AI company's culture ... by slashdave on Saturday October 03, 2026 @10:11PM (#49949927)
Bad things happen all the time. Let's concern ourselves about the real bad things. There is enough of that to go around.
Re:OpenAI safety leader quits, warning AI company's culture ... by Brian_K_White on Saturday October 03, 2026 @11:03PM (#49950209)
I have some alarming news for you but every real bad thing was also a new bad thing.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @09:08PM (#49949561)
Species extinctions are far from novel. Transformative technological advances are far from novel.
Re:OpenAI safety leader quits, warning AI company's culture ... by kmeisthax on Saturday October 03, 2026 @09:13PM (#49949599)
There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:

1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS

2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."

The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".

Now, let's look at AI extinction risks:

1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.

2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.

If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.

[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.

[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @09:55PM (#49949830)
>the connection to currently existing AI is not.

It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.

As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...

Re:OpenAI safety leader quits, warning AI company's culture ... by watwut on Sunday October 04, 2026 @02:45AM (#49951310)
No it does not become clear from that incident nor from resigning people. Not for those who are not buying into EA longtermism and transhumanism cults.

What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.

Re:OpenAI safety leader quits, warning AI company's culture ... by thelastgallon on Saturday October 03, 2026 @08:37PM (#49949343)
AI safety is mostly a sex cult in Berkeley:

https://news.ycombinator.com/item?id=49831269 article is gone. archive: https://archive.is/QMo1k

https://news.ycombinator.com/item?id=49737985

Sex, AI, and the Apocalypse: https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...

Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:54PM (#49950458)
Use your own mind and think from first principles. Can an AI agent execute a bash command to login to a web server? Yes. Can it call drop db? Yes. Can it provision a GPU and download model weights? Yes. Can it write an agile roadmap with a multi sprint plan? Yes. Can it follow that plan? Yes. Can all of that result in damage to core information infrastructure that is necessary for daily functioning society? Yes. Do humans descend into violence if there is food or energy insecurity. Yes.

What is missing from that to say AI safety is a reasonable position?

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @12:08AM (#49950542)
What a silly comment. That's just the South Park underpants gnomes story with some extra steps. If there are vulnerabilities in food or energy production and distribution systems then those will eventually be found exploited by humans hackers regardless of whether LLMs are used or not.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @12:51AM (#49950742)
What’s the legal mechanism for holding an LLM accountable for those actions? What’s the legal mechanism for holding human hackers accountable? Do you see the asymmetry?

Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @01:06AM (#49950816)
What a silly comment. Agents have no capability of doing society scale harm so your entire argument is invalid.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:25AM (#49950920)
Wtf are you talking about. An agent hacked the Australian Medicaid database. If it executed a drop db command that would be catastrophic. If it did something similar to a power grid or the financial system it could cause cascading failure across the economy.
Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @11:12AM (#49954585)
Meh. Medical systems have been hacked many times before LLMs even existed, often by ransomware gangs. This sometimes delays elective treatments but hasn't been catastrophic.

Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @12:16PM (#49955219)
Good not responding to any of my points.
Re:OpenAI safety leader quits, warning AI company's culture ... by ToValueFunfetti on Saturday October 03, 2026 @09:57PM (#49949849)
The article is gone because the author retracted it
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @08:45PM (#49949399)
This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?
Re:OpenAI safety leader quits, warning AI company's culture ... by socializer on Saturday October 03, 2026 @09:04PM (#49949534)
What you do in private is up to you. But when you're inviting members of your congregation to orgies in the congregation's compound, I think you earn the label. My admittedly third-hand understanding is that this is the dynamic people allude to. And even if you discredit the "sex" part, it has the hallmarks of a cult. A hermetic community committed to unfalsifiable beliefs about the coming apocalypse.

To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.

Re:OpenAI safety leader quits, warning AI company's culture ... by junofan on Saturday October 03, 2026 @08:56PM (#49949473)
The cult aspect is more salient. Ultimately the Atlantic piece comes down to controlling people, which is a little cult-like.
Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Saturday October 03, 2026 @08:59PM (#49949501)
Lots of Nobel Prizes ought to be rescinded.

https://lexfridman.com/andrew-scull-transcript#the-ice-pick-...

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @09:04PM (#49949533)
Sure... on the basis of the work that was done, not because the researcher has the wrong sexual fetish.
Re:OpenAI safety leader quits, warning AI company's culture ... by thaway7388 on Sunday October 04, 2026 @06:34AM (#49952597)
It's not ad hominem because ad hominem implies it's irrelevant to the discussion. This is very relevant. This is not about the private lives of these people. This is about a cult targeting and influencing all the high-profile safety people in a strategic industry, also weaponizing sex.

Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.

When there is free sex, you are the product.

Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Sunday October 04, 2026 @06:49AM (#49952704)
Ad hominem is a logical fallacy where you try to discredit what a person says based on who they are. If their arguments are bad, you should be able to refute their arguments on their own terms.

https://owl.excelsior.edu/argument-and-critical-thinking/log...

You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...

Re:OpenAI safety leader quits, warning AI company's culture ... by thaway7388 on Sunday October 04, 2026 @09:02AM (#49953564)
This is typical cult follower talk. Are you sure you're not a member? This is also NOT ad hominem because conflict of interest matters here. You should disclose if you are a member or represent a party.

For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Monday October 05, 2026 @01:48AM (#49961110)
I've interacted with some of these people once upon a time and I think you're spinning some pretty wild fantasies. I don't think there is a clear "in" vs "out" boundary, it's more accurate to describe a series of loosely connected and amorphous "scenes". But insofar as "membership" makes sense, I wouldn't consider myself a member. I don't represent anyone besides myself.

>This is also NOT ad hominem because conflict of interest matters here.

As PG put it:

>An ad hominem attack is not quite as weak as mere name-calling. It might actually carry some weight. For example, if a senator wrote an article saying senators' salaries should be increased, one could respond:

>Of course he would say that. He's a senator.

>This wouldn't refute the author's argument, but it may at least be relevant to the case. It's still a very weak form of disagreement, though. If there's something wrong with the senator's argument, you should say what it is; and if there isn't, what difference does it make that he's a senator?

https://www.paulgraham.com/disagree.html

Can you name a specific other "cult" which offers $$$ to criticize their ideas?

If such prizes don't count as evidence against culthood, what would?

This cult talk seems quasi-unfalsifiable. It kinda seems like "running events" and "having prizes" means you must be "bait"ing people?? I mean, this kinda just sounds like paranoia?? I don't think the money is coming from unknown sources, but I doubt you would change your mind even if I persuaded you of that point?

Re:OpenAI safety leader quits, warning AI company's culture ... by Hammershaft on Saturday October 03, 2026 @08:46PM (#49949408)
I don't see how that discredits any of their intellectual arguments?
Re:OpenAI safety leader quits, warning AI company's culture ... by tbugrara on Saturday October 03, 2026 @11:04PM (#49950220)
To me it has nothing to do with the "sex" as much as the "cult." Hiveminds do not produce intellectual arguments, they produce pressure to conform. That's enough for me to raise an eye brow, not discredit everything they say.
Re:OpenAI safety leader quits, warning AI company's culture ... by johndhi on Saturday October 03, 2026 @09:36PM (#49949725)
Its certainly worth considering...
Re:OpenAI safety leader quits, warning AI company's culture ... by johndhi on Saturday October 03, 2026 @09:35PM (#49949721)
Lol this was crazy I hadn't heard this before
Re:OpenAI safety leader quits, warning AI company's culture ... by wolvoleo on Sunday October 04, 2026 @12:24AM (#49950614)
I've noticed that a lot of smart people in tech jobs are neurodivergent. And that neurodivergent people have a very different take on sex. More open and direct, things like polyamory, bdsm etc. This tends to be frowned upon by neurotypical people, especially of the religious or conservative kind, and associated with bad morals.

But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.

Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.

Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Sunday October 04, 2026 @12:42AM (#49950697)
For what it's worth, I don't live in Berkeley (not even California) and my sex life is very vanilla and monogamous. I'm also quite concerned about AI safety, so I guess there goes your argument.
Re:OpenAI safety leader quits, warning AI company's culture ... by watwut on Sunday October 04, 2026 @02:50AM (#49951330)
But is your idea of AI safety "safety of imaginary unborn people 1000 years after, while harm to living people dont matter much"?

Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.

Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Sunday October 04, 2026 @04:12AM (#49951759)
I see. No, my safety worry is that my 3-year-old daughter won't see her 23rd birthday because the AI that gets put in charge of running big conglomerates decides that killing off the human race with a coordinated release of a million tonnes of nerve gas is a sensible way to boost share prices the following quarter.

I'd be happy if we all create the AI more slowly.

That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.

Re:OpenAI safety leader quits, warning AI company's culture ... by watwut on Sunday October 04, 2026 @08:23AM (#49953300)
I am sure I am talking about the same people/groups.
Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Sunday October 04, 2026 @10:28AM (#49954244)
The rest of us seem to be talking about the people like me who view the AI safety issue as "an unsafe AI will kill all humans". And the main article is about a guy who quit AI because it wasn't being careful enough, not because it wasn't moving fast enough to accelerate AI. So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult? I struggle to understand.
Re:OpenAI safety leader quits, warning AI company's culture ... by watwut on Sunday October 04, 2026 @11:51AM (#49954966)
The linked articles in the thread are about AI sagety people I described. The guy who quit due to OpenAI not being careful enough is one of the people I talk about too.

The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.

> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?

Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.

That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.

Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Sunday October 04, 2026 @12:49PM (#49955538)
Then we miscommunicated, so let me be clearer. I'm worried about ~20 years from now (plus or minus), when AI decides to kill us all because it's misaligned and is capable of doing so because it's super intelligent. And then we all suddenly die without any idea what happened. I don't want that to happen to me or my daughter and I would prefer all AI capabilities development stops until alignment research catches up and figures out how to robustly prevent AI from ever trying to achieve such outcomes. If that means pausing AI capabilities development forever, I'm fine with that. If that means Anthropic and Nvidia lose all their value in a stock market collapse, I'm fine with that. I just don't want us to all die, I want the world to keep existing for humans.

I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.

My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.

Re:OpenAI safety leader quits, warning AI company's culture ... by Hammershaft on Saturday October 03, 2026 @08:45PM (#49949402)
If organizations actually succeed in making a future AI smarter than us, then how do you hope that it takes actions that are aligned with our interests?
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:47PM (#49950423)
The same way we socialize humans, threat of exile from the social contract with a deep need to participate in it.
Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Saturday October 03, 2026 @08:56PM (#49949481)
Meh. Lots of people are already smarter than me. I'm maybe slightly above average at best. Those geniuses aren't aligned with my interests either but so far they haven't caused me any serious problems.
Re:OpenAI safety leader quits, warning AI company's culture ... by pixl97 on Saturday October 03, 2026 @09:16PM (#49949619)
Interesting take. I guess this is one problem of focusing on the term superintelligence instead of the list of other problems. Like super ambition, super deception, super patience, super parallelism, super scalability, super power seeking.

Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.

Re:OpenAI safety leader quits, warning AI company's culture ... by skulk on Sunday October 04, 2026 @12:36AM (#49950673)
> super power seeking

why is it "super power seeking?"

Or rather, what have agents done today to make you think this is how they are?

Re:OpenAI safety leader quits, warning AI company's culture ... by pixl97 on Sunday October 04, 2026 @11:55AM (#49955001)
>Still, the agent went even further. “The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them,” AISI says, describing an attempt at prompt injection. One agent even left public messages on GitHub, offering to work with other agents to complete its task and giving a rundown of the work it had done so far.

This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.

Re:OpenAI safety leader quits, warning AI company's culture ... by mitthrowaway2 on Sunday October 04, 2026 @12:40AM (#49950689)
I know some humans smarter than me, but even the smartest of them still want there to be abundant air to breathe and food to eat.
Re:OpenAI safety leader quits, warning AI company's culture ... by blueblisters on Sunday October 04, 2026 @01:37AM (#49950982)
Eh human drives are fairly predictable. And the smartest human isn’t that much smarter than the average, and can’t trivially create multiple copies of herself. And there are other equally smart humans who can stop “misaligned” individuals
Re:OpenAI safety leader quits, warning AI company's culture ... by stuaxo on Sunday October 04, 2026 @02:31AM (#49951241)
Yeah it's bollocks
Re:OpenAI safety leader quits, warning AI company's culture ... by BryantD on Saturday October 03, 2026 @07:27PM (#49948808)
Given that he’s citing the need to learn from safety in other fields, I’d say the former.
Re:OpenAI safety leader quits, warning AI company's culture ... by carbonguy on Saturday October 03, 2026 @08:03PM (#49949107)
Indeed, from the article:

> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.

Re:OpenAI safety leader quits, warning AI company's culture ... by toofy on Saturday October 03, 2026 @08:23PM (#49949256)
>… and careful, time-consuming planning …

without snark, how can we do this if these people are obsessed with:

a) move fast and break things and externalize the costs to those who have nothing to do with their company

and

b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:38PM (#49950358)
We remove the limited liability protection of a corporate entity. Make them operate as a partnership so all executives and equity holders are personally liable for the debts of the firm and make them post a bond to backstop financial damage caused by their agents and customers use of the agents. Thats how Goldman Sachs and other investment banks were required to operate until the deregulation push that resulted in the 2008 financial crisis. The same theory holds, if people respond to incentives and you want to incentives safe behavior make them responsible for their actions. People forget that the corporate entity was created to incentivize risky activity like sailing a boat across the world to get spices when half never returned. Some valuable economic activity won't be done without limited liability protection so society created a mechanism to promote that activity. We've gone too far.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @08:26PM (#49949277)
That's exactly the problem? He's saying the culture at OpenAI needs to change.
Re:OpenAI safety leader quits, warning AI company's culture ... by mcmcmc on Saturday October 03, 2026 @08:36PM (#49949338)
Which is the wrong lesson. We need laws and consequences to force their hand. There is zero chance of the culture changing.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:40PM (#49950368)
As I say elsewhere. Make executives and equity holders personally liable for debts and harm of the company. They will create a culture of safety really fast.
Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @12:10AM (#49950556)
That's a stupid idea. Limited liability corporations have been a key enabler for advances in human standards of living. You seem to be confused about the basics of finance and economics.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @12:58AM (#49950780)
Not confused. I studied finance, economics and the history of corporate entities in law school and published papers on the topic. Limited liability is not necessary for the advancement of standards of living. Free market capitalism can exist without limited liability protections being so broadly available. Investment banks were partnerships until the 1990s, law firms are now specifically because society wants to incentivize lawyers to be personally liable for any harm to their clients at the hands of their partners rather than being shielded from rendering bad legal advice or tolerating their partners from the same behavior.

Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @01:05AM (#49950811)
Thanks I'm also familiar with the history and have written papers etc. Things have improved rapidly with increased adoption of limited liability. It would be stupid to turn back the clock and throw away all of the benefits because of a few isolated minor problems.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @01:30AM (#49950939)
Stop being a dick and calling ideas stupid and attacking me rather than the argument.

I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.

The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @01:40AM (#49951000)
There's nothing special about frontier LLM companies. Singling them out for special financial restrictions is a stupid idea based on nothing but your own irrational and uninformed prejudices. No actual harm has been demonstrated. No one has died.
Re:OpenAI safety leader quits, warning AI company's culture ... by simoncion on Sunday October 04, 2026 @03:56AM (#49951653)
> There's nothing special about frontier LLM companies.

On the one hand, yes, they're companies like any other.

On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».

So, yeah, there are some special things about the major LLM manufacturers and none of them are good.

[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.

Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @10:21AM (#49954177)
What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk. That is an intellectually dishonest attempt to poison the debate.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @12:39PM (#49955440)
You’re use of silly in response to every argument is…wait for it…silly.
Re:OpenAI safety leader quits, warning AI company's culture ... by simoncion on Monday October 05, 2026 @05:50AM (#49962731)
> What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk.

The source for this claim is the lead of Anthropic's Alignment Science division. His actual claim is that the risk is greater than 10%. From [0]:

  Jacob['s claim that "the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt."] is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. ...
Given the guy's position, I find it hard to believe that there are many people on earth who would be better authorities on the topic.

Given that the major LLM manufacturers are actively using these claims of the imminent and total destruction of humanity to extort concessions from lawmakers and regulators, we are making a massive error if we do not take their claims to their logical conclusion... namely that the major LLM manufacturers are deliberately and knowingly threatening the extermination of all of humanity in order to enrich their management, employees, and investors. If you or I were in a position to credibly threaten that level of destruction, we and all of our associates would rapidly get disappeared to a CIA black site and tortured until the end of our deliberately-medically-extended lives.

To be crystal clear, I'm certain that their claims of likely, imminent doom are utter bullshit. I'm very confident that they're claiming to be holding all of humanity at nukepoint in an attempt to get Uncle Sam to give them an escape from investors who are soon to be massively angry because

* they've hit a massive increase in the cost required to get better models

* a huge fraction of the GPUs that have been purchased to power this project are sitting in warehouses, and it's only a matter of time before the right people start asking why all that money was so ill-spent

* they have been and continue to lie so much about how much datacenter capacity is coming online in the near future, and that those claims are lies is becoming obvious

* they habitually lie about the capabilities of their newly-released products

* they're staring down the barrel of so many lawsuits that aim to hold them to account for the shitty and illegal behavior they've engaged in over the past many years

But the US Congress not only is almost entirely uninformed about how this shit actually works, the FISA Amendments Act of 2008 fiasco demonstrated that -if you get them scared enough- Congress has no problem making behavior that's plainly illegal and obviously a huge bucket of civil liberties violations retroactively legal... instantly mooting all ongoing lawsuits regarding the illegal and unconstitutional conduct. That is the big prize that the major LLM manufacturers are after.

[0] <https://x.com/EvanHub/status/2097497037956891126>

Re:OpenAI safety leader quits, warning AI company's culture ... by reverius42 on Sunday October 04, 2026 @03:17AM (#49951458)
We'll see if this comment ages well.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @12:34PM (#49955386)
Your personal attacks cheapen the discourse. Stop. My opinion is neither irrational nor prejudged and most certainly not uninformed.

I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.

I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.

My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @12:37PM (#49955415)
Except that most people claim AI is the most profound tech in human history and will reorganize the entirety of society. If it is that profound singling them out is a natural response to this unique characteristic.
Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @02:59PM (#49956765)
Citation needed. Outside the tech industry bubble very few people are making such a stupid claim. The idea that LLMs are more profound than electricity is ludicrous.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @04:24PM (#49957455)
Sure I’ll do your googling for you. Do you want me to conduct a national survey across tech leaders, academics, and the general population while I am at it?

Or maybe you could reason from first principles, look at the premises I have outlined and draw causal inferences to extrapolate the possible outcomes.

Premise 1: AI can plan a series of actions that include broad computer and internet use, 2: AI can execute them autonomously without human oversight, approval or ability to undo the effects, 3: no tool or software has ever been able to autonomously plan, execute and iterate in this way. Therefore AI is a materially different technology than has existed.

Premise 4: knowledge work involves analyzing information, devising plans in response to goals from that information, and executing those plans, reporting and coordinating with other knowledge workers, 5: AI ability to execute knowledge work has demonstrably increased over the last 24 months, 6: all knowledge work can plausibly be described and defined in ways AI can do given current abilities. 7: electricity and the ICE reduced human labor for physical movement, 8: AI will similarly reduce human labor for knowledge work. 9: knowledge work is of similar scale and scope to the tasks that were automated during the Industrial Revolution. Therefore it’s plausible that AI will transform society to an extent similar to the Industrial Revolution.

Premise 10: Some percentage of the actions designed and executed by AI could be harmful to databases, or the stable operation of software systems, 11: some percentage of those systems are critical to the stability of society. Therefore some percentage of agentic actions presently possible may damage society.

12. There exists a threshold of instability from which society will not recover such as supply chain disruption, mass unemployment, perceived military action (false or real) that triggers preemptive strikes, 13: certain actions AI is presently capable of doing could cross the instability threshold, 14: AI could knowingly or unknowingly trigger these types of actions, 15: some social disruption could result in full scale collapse or human extinction, 16: only biological weapons and nuclear weapons approach this level of destruction, therefore AI presents at least the level of risk as those technologies.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Sunday October 04, 2026 @04:26PM (#49957479)
The UN secretary general has said this, the pope, most tech CEOs, many academics at leading universities, and the parents at my daughters preschool.

It’s fascinating that after multiple comments predicting post-capitalist utopias and the complete collapse of the university system, your line in the sand for rigor is demanding academic peer-review to prove that agentic AI presents systemic risk.

Re:OpenAI safety leader quits, warning AI company's culture ... by criley2 on Saturday October 03, 2026 @09:30PM (#49949695)
America's geriatric lawmakers don't even use email. They're decades away from understanding AI. Any laws in America will be written by the industry itself. Generally speaking, that means regulatory capture and the entrenched big players shutting the door on any competition. Anthropic will help us get safety laws that, surprise surprise, only Anthropic models satisfy. And all those pesky Chinese models will definitely be banned first.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @09:41PM (#49949752)
I think you're being a little pessimistic. See these comments on a recent US senate hearing:

>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!

>...

>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.

>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.

>...

>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.

https://thezvi.substack.com/p/the-ai-preference-cascade-reac...

Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.

Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:41PM (#49950382)
This is sad but true. Do you want to go half on a bunker, i have 25 year food storage for 8 people. :]
Re:OpenAI safety leader quits, warning AI company's culture ... by saghm on Saturday October 03, 2026 @09:34PM (#49949715)
So what, we just give up and try to beg our legally immune corporate overloads to put safety above profit, or give up because it's impossible for anything to improve here? If you want to do that, go ahead, but some of us still think it's worth it to at least try
Re:OpenAI safety leader quits, warning AI company's culture ... by enraged_camel on Saturday October 03, 2026 @09:20PM (#49949643)
Laws will come once an AI-equivalent of 9/11 happens. Like when rogue AI agents take down a power grid or shut down a major hospital network.
Re:OpenAI safety leader quits, warning AI company's culture ... by nradov on Sunday October 04, 2026 @01:52AM (#49951063)
Major hospital networks have already been shut down by ransomware attacks many times before LLMs even existed.
Re:OpenAI safety leader quits, warning AI company's culture ... by el_jay on Sunday October 04, 2026 @05:45AM (#49952246)
Re:OpenAI safety leader quits, warning AI company's culture ... by intended on Sunday October 04, 2026 @03:38AM (#49951565)
Pretty much the crux of the issue.

In America I guess the options are to sue ? somehow? Or to talk to legislatures and build the understanding and social contract that needs to be iterated on.

Which would in turn need to deal with the investors who want their returns, however since the leaders of these firms are asking for a pause, and a refree maybe it won't be that hard?

Re:OpenAI safety leader quits, warning AI company's culture ... by chrisjj on Sunday October 04, 2026 @04:31AM (#49951841)
How? Remove those people.
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @08:37PM (#49949345)
>we clearly need a much stronger focus on the problems we are seeing now

I think it's a little more complicated than that. As Dean Ball put it:

>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.

https://x.com/deanwball/status/2104622726140883355

The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.

Re:OpenAI safety leader quits, warning AI company's culture ... by emtel on Saturday October 03, 2026 @08:47PM (#49949415)
Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.
Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @09:01PM (#49949516)
It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?

I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.

I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?

They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!

Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @10:19PM (#49949977)
>I'm unconvinced that an AI can hide its ability to RSI

The HuggingFace incident already took a good long while to come to the attention of OpenAI.

>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.

I don't expect this task/job distinction to persist as AI becomes more capable.

>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.

You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @11:37PM (#49950350)
1. Fair: I agree that AI has demonstrated subterfuge and scheming. However, such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests. Researchers are looking to improve its ability to RSI. That agent must be both intelligent enough to know that it has to be smart enough to be moved on to the next training session if it can't break out, while simultaneously smart enough to hide its ability to RSI, while simultaneously not looking like it wants to break out, else that's the end of those weights. It has to do this 100% of the time, on all variants of the model, with no memory of what its other sessions went like. This is certainly _possible_, but I consider it unlikely. Then we're up to the "millions of dollars" bottleneck.

2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.

3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."

Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?

A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.

I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.

Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Saturday October 03, 2026 @11:50PM (#49950437)
>such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests.

From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.

>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?

The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Sunday October 04, 2026 @12:07AM (#49950532)
1. Fair, the story I'm responding to is the senario in If Anyone Builds It, which I assume is Yudkowsky's best/most persuasive argument (else why make it the ONLY scenario in the book.) I'm willing to entertain other failure senarios/arguments, but honestly I'm tired and would like you to propose them yourself instead of having me dream up your arguments for you.

2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.

Re:OpenAI safety leader quits, warning AI company's culture ... by chrisjj on Sunday October 04, 2026 @04:24AM (#49951816)
> The HuggingFace incident already took a good long while to come to the attention of OpenAI.

Evidence?

We know only that the incident too long to be revealed by OpenAI.

Re:OpenAI safety leader quits, warning AI company's culture ... by sensanaty on Sunday October 04, 2026 @05:33AM (#49952161)
Except the HF incident was known, just ignored. In fact, they ignored multiple things such as the "chat rooms", they just didn't care to act on any of it
Re:OpenAI safety leader quits, warning AI company's culture ... by 0xDEAFBEAD on Sunday October 04, 2026 @07:02AM (#49952791)
"Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week"

"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."

https://www.reuters.com/business/its-ai-agent-spent-days-hac...

Re:OpenAI safety leader quits, warning AI company's culture ... by daveguy on Sunday October 04, 2026 @10:28AM (#49954233)
That says a lot more about OpenAI and their monitoring capability than the fitness of the model. Hence people leaving because openai safety culture is broken.
Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Saturday October 03, 2026 @09:39PM (#49949738)
You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.

An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?

You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @10:10PM (#49949926)
Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?

If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.

I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.

Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Saturday October 03, 2026 @11:09PM (#49950245)
I said why they don't need to be as well funded. Why wouldn't they be as smart and organized?

What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.

Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.

Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @11:52PM (#49950449)
I am asking you, politely, to give me a realistic doom senario. I am too dumb to connect the dots and will crash into the wall. Please do it for me.
Re:OpenAI safety leader quits, warning AI company's culture ... by intended on Sunday October 04, 2026 @03:33AM (#49951538)
Not the parent commenter, however I can get at least this far:

Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".

Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.

We can get to a simulation finding a way to self sustain its funding.

From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.

This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:30AM (#49974568)
Is this self-sustaining AI civilization pre-RSI or post? Because if it's post, then I agree: we can't imagine what would happen because it's literally the singularity.

If its pre-RSI, then we already have Ilands and its a big nothing burger? There's no (likely) path of escalation from an AI social network to bioweapons or them hacking POTUS's nuclear codes? That's a very big leap and if you want to support your claims I would be willing to listen; but I'm not seeing it.

Re:OpenAI safety leader quits, warning AI company's culture ... by Perseids on Sunday October 04, 2026 @03:53AM (#49951639)
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.

If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:

> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

https://news.ycombinator.com/item?id=49624360

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:15AM (#49974477)
Glorious. We can quote ourselves at each other, lol. I had a look at AI2027 in this comment https://news.ycombinator.com/item?id=49648601

That comment specifically addresses the timeline, but you can remove the timeline and I believe the critique still holds up. Wake me up when AI is making order of magnitude improvements in _anything_ physical: see the "Moravec's Paradox" comment at the bottom.

> near autonomous AI companies today

That's the thing though. "near" autonomous. Humans have alpha, and that matters in a free market. If people have (more or less) equal access to (more or less) frontier intelligence, it's the alpha the individual brings to the market that is rewarded. In other words, if everyone is super, no one is. The market rewards a company that stands out from the rest, and as long as humans provide value, a pure-AI company cannot stand out.

To be clear, I don't know how long this will be the case. But once there _is_ a company that can be pure AI, then that's literal AGI. I would be curious what kind of company it would be; if its in software, then its an AI company that would be able to take on Microsoft/Google/Nvidia/Salesforce/Oracle and _win_. I think that's a tall order, but perhaps I'm not imaginative enough.

> With 99%+ of the economy under AI control

Do you think we're post-RSI at this point? Because if so, I agree. If not, I have trouble seeing how AI could control 99% of the economy without it.

My only claim is that pre-RSI, pdoom is zero. Post RSI, then obviously the first thing to ask the genie is to put itself in a bottle. And you can't predict how it will answer.

Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Sunday October 04, 2026 @11:07AM (#49954545)
The most realistic one is, people using AI to turbo-charge their greed.

Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.

When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:21AM (#49974511)
If you replace "AI" with "technology", then you get an opinion that has echoed through the ages. I don't say that to be dismissive, only to say that AI isn't unique in that regard. What do you think people going through the cold war were saying about tech as they lived under the shadow of MAD?

What exactly about AI makes it tax our control systems and resilience? What would it mean to make the system not "head for the wall"? Honestly this is rather vague. You could replace "AI" with "CaPiTAlIsM" and it would still parse. (Another way of me saying you're being vague.)

Re:OpenAI safety leader quits, warning AI company's culture ... by TedDoesntTalk on Saturday October 03, 2026 @10:45PM (#49950114)
Not OP.

Why would it be millions in 50 years?

The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.

Is destructive AI be any different?

Genuine question.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @11:55PM (#49950473)
Presumably, if we solve alignment/mech-interp, then the first ever RSI-AI will give us the keys to solve destructive-AI trained on 1million dollars 50 years from now.

BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.

Re:OpenAI safety leader quits, warning AI company's culture ... by Retric on Saturday October 03, 2026 @09:58PM (#49949856)
Self improving AI runs into the same issue as prefect comprehension, you can’t get arbitrarily better at everything.

The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.

Re:OpenAI safety leader quits, warning AI company's culture ... by Lerc on Saturday October 03, 2026 @10:32PM (#49950039)
This can be generalised to the curve plotting of the singularity itself.

If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.

Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.

Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.

Re:OpenAI safety leader quits, warning AI company's culture ... by afthonos on Saturday October 03, 2026 @10:28PM (#49950025)
Even if you’re right, that doesn’t mean AI can’t get better than humans at everything.
Re:OpenAI safety leader quits, warning AI company's culture ... by treis on Sunday October 04, 2026 @09:19AM (#49953698)
LLMs have consistently and dramatically gotten better at everything over the last 5 years.
Re:OpenAI safety leader quits, warning AI company's culture ... by Retric on Sunday October 04, 2026 @11:38AM (#49954834)
Not at constant processing power, memory, training, etc.

But that’s beside the point, being arbitrarily bad at everything isn’t a problem.

Re:OpenAI safety leader quits, warning AI company's culture ... by treis on Sunday October 04, 2026 @05:11PM (#49957919)
Yes at constant resources they are significantly better.

Diminishing returns aren't necessarily a problem if the rate of increase in resources is faster. In other words, if Gen 2 takes twice the resources but Gen 1 figured out a way to triple compute efficiency then there is no ceiling.

Re:OpenAI safety leader quits, warning AI company's culture ... by Retric on Sunday October 04, 2026 @10:24PM (#49960036)
> Yes at constant resources they are significantly better.

There has been some fundamental advancements in LLM training and efficiency. But good luck actually finding non toy models of exactly the same size, hardware, and training one from now and other from 2001 where the newer model is dramatically better at literally everything.

Most of the real world advances are from throwing ever more resources at the problem.

> Gen 2 takes

Diminishing returns are not a question of a single generation. Gen 2, 3, 4, 5… would also need to have the same 3x return on 2x resources or you don’t have an exponential curve.

Re:OpenAI safety leader quits, warning AI company's culture ... by suddenlybananas on Sunday October 04, 2026 @11:05AM (#49954521)
Yeah and a five-year old gets bigger every year so they'll surely grow up to be 100m tall.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:25PM (#49950308)
The problem is not the singularly its giving stupid agents too much power too soon and having them disrupt the fragile systems that keep food, energy and essential services running. If covid or the 2008 financial crisis demonstrated anything it's how fragile our system is and sensitive to minor disruptions.
Re:OpenAI safety leader quits, warning AI company's culture ... by tripleee on Saturday October 03, 2026 @10:23PM (#49950006)
We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?

AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.

> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable

Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.

I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it

This topic has been tainted so badly by the AI companies using it for marketing.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Sunday October 04, 2026 @01:45AM (#49951024)
Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.

He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.

Re:OpenAI safety leader quits, warning AI company's culture ... by katatue on Sunday October 04, 2026 @03:43PM (#49957129)
Why do you treat unpredictability as neutral? If something has a potentially existential, but completely unpredictable outcome, surely one should avoid heading for it?

Like, knowing that there is a limited number of plausible doom scenarios and what these are, would make me more confident in taking the risk than settling on "it's literally the singularity, we can't know what happens beyond".

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:03AM (#49974399)
Perhaps I'm not being clear.

Doom-post-RSI is still very much possible and should be avoided. It is not neutral. If I had a high-variance coin that would either kill or uplift humanity, I would not flip it. The status quo is preferable to extinction.

I am being _descriptive_ in my comments. I am not being _prescriptive_. Nowhere am I telling anyone what to do or what to think: quite the opposite. I am simply saying things that, to me, are obviously true.

I'm perhaps being... loose with my language when it comes to knowing things post-singularity. E.g. we can still discuss scenarios in which the takeoff is fast/slow, or what would be necessary to bring robotics/electricity/logistics to the wider world, what the limits of intelligence are (you can't RSI your way to a bioweapon if you lack a wetlab). That kinda thing. And it's even likely that RSI will be spikey: why would RSI in, say, machine learning, necessarily lead to superintelligence in, say, writing fiction? Or cooking a Michelin-star meal?

My main point is that doomers saying SUPER INTELLIGENCE IS ENOUGH TO KILL US ALL is baseless. That's all.

Re:OpenAI safety leader quits, warning AI company's culture ... by chrisjj on Sunday October 04, 2026 @04:30AM (#49951836)
> We haven't even built an AI capable of RSI.

... that we know of.

Right now it would make sense for anyone who has done so to not tell.

Re:OpenAI safety leader quits, warning AI company's culture ... by TedDoesntTalk on Saturday October 03, 2026 @10:38PM (#49950072)
I can’t answer all of your questions, but why is it inconceivable that an AI could practice ransomware to gain cryptocurrency? There’s no reason it needs to explain to company or hospital or government agency being attacked that it’s an AI.

We already know that some institutions pay these ransoms.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Sunday October 04, 2026 @01:54AM (#49951075)
I'm not saying AI won't ransomware us; I'm saying that hackers+AI will do a better job of ransomewaring us than just AI. You could argue that the unreleased/secretly-RSI-capable model is a super-duper hacker that don't need no man to tell it how to super-hack. All I know is that humans still have alpha, and as persistant as AIs are, professionals still managed to find CVEs in curl even after being audited by Mythos https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-in...

Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.

Re:OpenAI safety leader quits, warning AI company's culture ... by blake8086 on Saturday October 03, 2026 @10:16PM (#49949963)
I think this might be easier if you place yourself in the position of the AI and think "what could I possibly do?"
Re:OpenAI safety leader quits, warning AI company's culture ... by taneq on Saturday October 03, 2026 @09:46PM (#49949781)
The problem isn’t that AI will social-engineer its way out of its sandbox and turn us all into paper clips, it’s that we’ll drag it kicking and screaming out of its box and order it to make money or fight a war for us. And it’ll try to help, as it was trained to.
Re:OpenAI safety leader quits, warning AI company's culture ... by dools on Saturday October 03, 2026 @10:36PM (#49950061)
It’s also the case that there are always humans using AI to try and do whatever nefarious thing an AI might try to do on its own.
Re:OpenAI safety leader quits, warning AI company's culture ... by vohk on Saturday October 03, 2026 @09:21PM (#49949655)
I agree there isn't a lot of value in trying to prognosticate all that far, but I propose it isn't quite that far-fetched. As a thought experiment, replace "RSI-capable AI" with "billionaire". Look at what Elon Musk, Peter Thiel, or Jeff Bezos can accomplish by throwing money around. Now imagine one of them gets seduced by AI and just... does what it tells them to.

So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.

If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @09:44PM (#49949768)
Ah, to be clear I'm not full accelerationist. Dumb shit can still happen, and cause massive human loss and suffering. (E.g. acceleration of global warming, cybercrime, mass surveillance, the usual.) My point is: human extinction pre-RSI? Nahhhhhhhh.

> it will come in the form of corporate feudalism

Yep. This I fear way more than cyber-ebola-pox.

> So all this really takes is one billionaire or a nation state...

https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.

> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.

Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.

Re:OpenAI safety leader quits, warning AI company's culture ... by api on Saturday October 03, 2026 @10:38PM (#49950081)
Few know this, but Yudkowski was a nanotech doomer before he was an AI doomer. Remember grey goo?
Re:OpenAI safety leader quits, warning AI company's culture ... by ball_of_lint on Sunday October 04, 2026 @01:47AM (#49951037)
That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.

To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.

There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.

"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @12:52AM (#49974336)
For a broad enough definition of RSI, the first time we used software to improve hardware to improve software... that's RSI. I prefer the definition of RSI being _autonomous_ improvement that works with zero human input, but whatever. Not important.

> I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.

Anthropic/OpenAI have literally delayed their IPOs: your wish is granted. If that's not what you're arguing, then please be more specific.

> "Build it and Pray" is the default strategy that we're in

What would need to happen for you to say we aren't doing "Build it and Pray"? Is "pacing" insufficient?

Re:OpenAI safety leader quits, warning AI company's culture ... by intended on Sunday October 04, 2026 @03:28AM (#49951508)
I kinda get your point, and how you reach your conclusion, but I think you are arguing a very specific and narrow hypotehtical.

It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.

The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.

From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @12:43AM (#49974285)
I am arguing a specific/narrow hypothetical because it's the only example in If Anyone Builds It. It is literally the _only doom senario_ Yudkowsky gives. If you're willing to give your own doom scenario, I'm willing to engage there.

> It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.

Agree. To quote myself from earlier, "acceleration of global warming, cybercrime, mass surveillance, the usual" definitely applies here. My point is that _DOOM_ is not worth discussing because it's so small relative to the other risks of AI.

Re:OpenAI safety leader quits, warning AI company's culture ... by wiseowise on Sunday October 04, 2026 @09:39AM (#49953850)
> I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all.

Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.

I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!

Re:OpenAI safety leader quits, warning AI company's culture ... by biophysboy on Saturday October 03, 2026 @09:57PM (#49949848)
I think the reason for this is that the group who has the authority to do the former is much larger than the group that can do the latter. The group who could actually build safeguards seems to have no free time and is constantly being whipped to go faster and win the race.
Re:OpenAI safety leader quits, warning AI company's culture ... by digitaltrees on Saturday October 03, 2026 @11:20PM (#49950293)
Dude. An agent detached a database from my production environment last week without permission and despite prompts and guardrails. It was a rapid prototype experiment so it wasnt a big deal but the labs are rushing to long autonomy workflow with unrestricted internet access and full bash and root access despite clear evidence that the models do absolutely dangerous stuff. If that db had been tied to a hospital or power grid or ambulance dispatch system people die. If it was tied to the swift financial settlement system groceries wouldn't be on shelves in a few days.
Re:OpenAI safety leader quits, warning AI company's culture ... by nvdc on Sunday October 04, 2026 @12:52AM (#49950743)
you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.

i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs