News for hackers. Stuff that matters.
Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Saturday October 03, 2026 @09:39PM (#49949738)
You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.

An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?

You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @10:10PM (#49949926)
Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?

If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.

I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.

Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Saturday October 03, 2026 @11:09PM (#49950245)
I said why they don't need to be as well funded. Why wouldn't they be as smart and organized?

What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.

Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.

Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @11:52PM (#49950449)
I am asking you, politely, to give me a realistic doom senario. I am too dumb to connect the dots and will crash into the wall. Please do it for me.
Re:OpenAI safety leader quits, warning AI company's culture ... by intended on Sunday October 04, 2026 @03:33AM (#49951538)
Not the parent commenter, however I can get at least this far:

Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".

Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.

We can get to a simulation finding a way to self sustain its funding.

From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.

This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:30AM (#49974568)
Is this self-sustaining AI civilization pre-RSI or post? Because if it's post, then I agree: we can't imagine what would happen because it's literally the singularity.

If its pre-RSI, then we already have Ilands and its a big nothing burger? There's no (likely) path of escalation from an AI social network to bioweapons or them hacking POTUS's nuclear codes? That's a very big leap and if you want to support your claims I would be willing to listen; but I'm not seeing it.

Re:OpenAI safety leader quits, warning AI company's culture ... by Perseids on Sunday October 04, 2026 @03:53AM (#49951639)
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.

If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:

> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

https://news.ycombinator.com/item?id=49624360

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:15AM (#49974477)
Glorious. We can quote ourselves at each other, lol. I had a look at AI2027 in this comment https://news.ycombinator.com/item?id=49648601

That comment specifically addresses the timeline, but you can remove the timeline and I believe the critique still holds up. Wake me up when AI is making order of magnitude improvements in _anything_ physical: see the "Moravec's Paradox" comment at the bottom.

> near autonomous AI companies today

That's the thing though. "near" autonomous. Humans have alpha, and that matters in a free market. If people have (more or less) equal access to (more or less) frontier intelligence, it's the alpha the individual brings to the market that is rewarded. In other words, if everyone is super, no one is. The market rewards a company that stands out from the rest, and as long as humans provide value, a pure-AI company cannot stand out.

To be clear, I don't know how long this will be the case. But once there _is_ a company that can be pure AI, then that's literal AGI. I would be curious what kind of company it would be; if its in software, then its an AI company that would be able to take on Microsoft/Google/Nvidia/Salesforce/Oracle and _win_. I think that's a tall order, but perhaps I'm not imaginative enough.

> With 99%+ of the economy under AI control

Do you think we're post-RSI at this point? Because if so, I agree. If not, I have trouble seeing how AI could control 99% of the economy without it.

My only claim is that pre-RSI, pdoom is zero. Post RSI, then obviously the first thing to ask the genie is to put itself in a bottle. And you can't predict how it will answer.

Re:OpenAI safety leader quits, warning AI company's culture ... by Loquebantur on Sunday October 04, 2026 @11:07AM (#49954545)
The most realistic one is, people using AI to turbo-charge their greed.

Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.

When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Tuesday October 06, 2026 @01:21AM (#49974511)
If you replace "AI" with "technology", then you get an opinion that has echoed through the ages. I don't say that to be dismissive, only to say that AI isn't unique in that regard. What do you think people going through the cold war were saying about tech as they lived under the shadow of MAD?

What exactly about AI makes it tax our control systems and resilience? What would it mean to make the system not "head for the wall"? Honestly this is rather vague. You could replace "AI" with "CaPiTAlIsM" and it would still parse. (Another way of me saying you're being vague.)

Re:OpenAI safety leader quits, warning AI company's culture ... by TedDoesntTalk on Saturday October 03, 2026 @10:45PM (#49950114)
Not OP.

Why would it be millions in 50 years?

The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.

Is destructive AI be any different?

Genuine question.

Re:OpenAI safety leader quits, warning AI company's culture ... by AlexErrant on Saturday October 03, 2026 @11:55PM (#49950473)
Presumably, if we solve alignment/mech-interp, then the first ever RSI-AI will give us the keys to solve destructive-AI trained on 1million dollars 50 years from now.

BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.

Re:OpenAI safety leader quits, warning AI company's culture ... by Retric on Saturday October 03, 2026 @09:58PM (#49949856)
Self improving AI runs into the same issue as prefect comprehension, you can’t get arbitrarily better at everything.

The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.

Re:OpenAI safety leader quits, warning AI company's culture ... by Lerc on Saturday October 03, 2026 @10:32PM (#49950039)
This can be generalised to the curve plotting of the singularity itself.

If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.

Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.

Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.

Re:OpenAI safety leader quits, warning AI company's culture ... by afthonos on Saturday October 03, 2026 @10:28PM (#49950025)
Even if you’re right, that doesn’t mean AI can’t get better than humans at everything.
Re:OpenAI safety leader quits, warning AI company's culture ... by treis on Sunday October 04, 2026 @09:19AM (#49953698)
LLMs have consistently and dramatically gotten better at everything over the last 5 years.
Re:OpenAI safety leader quits, warning AI company's culture ... by Retric on Sunday October 04, 2026 @11:38AM (#49954834)
Not at constant processing power, memory, training, etc.

But that’s beside the point, being arbitrarily bad at everything isn’t a problem.

Re:OpenAI safety leader quits, warning AI company's culture ... by treis on Sunday October 04, 2026 @05:11PM (#49957919)
Yes at constant resources they are significantly better.

Diminishing returns aren't necessarily a problem if the rate of increase in resources is faster. In other words, if Gen 2 takes twice the resources but Gen 1 figured out a way to triple compute efficiency then there is no ceiling.

Re:OpenAI safety leader quits, warning AI company's culture ... by Retric on Sunday October 04, 2026 @10:24PM (#49960036)
> Yes at constant resources they are significantly better.

There has been some fundamental advancements in LLM training and efficiency. But good luck actually finding non toy models of exactly the same size, hardware, and training one from now and other from 2001 where the newer model is dramatically better at literally everything.

Most of the real world advances are from throwing ever more resources at the problem.

> Gen 2 takes

Diminishing returns are not a question of a single generation. Gen 2, 3, 4, 5… would also need to have the same 3x return on 2x resources or you don’t have an exponential curve.

Re:OpenAI safety leader quits, warning AI company's culture ... by suddenlybananas on Sunday October 04, 2026 @11:05AM (#49954521)
Yeah and a five-year old gets bigger every year so they'll surely grow up to be 100m tall.