The problem is that even if the majority of labs decide to slow down, there will be one lab, somewhere, with dark money moving forward. The military wouldn't let that chance pass by.

I think they will all conveniently "decide" to slow down because models are becoming crazy expensive, both per token and how much tokens they need to do anything meaningful. The difference between Sol and Astra is 150% pricing increase per token. The next one could be, I don't know, Orion, at $150 and it won't do anything meaningfully different than Astra. It's quite safe to say that they have ingested all human knowledge and because models are inherently just statistical generators, they are not expanding it, just rehashing what we already know anyway.

> there will be one lab, somewhere, with dark money moving forward

If this is really an existential threat, that's better than fifty of them.

No it's not. It's worse. Do you have any idea how much power that one lab would have?

How is worse?

If they manage to leap ahead they can effectively kill all competition, present and future and you'll be at their mercy.

So the solution is to make the company that can’t avoid sub-AGI models from going rouge to try RSI?

Also, AI performance has proven to be a function of power and compute, and right now most of the compute lies between two companies rushing towards RSI not because they are afraid someone is going to do this before but because they are the ones who wants to build it.

If the position is that one of these models will go out of control and destroy humanity then why would having two such models be safer than one?

Given that the arrival of this stuff has been compared to nuclear weapons already the question is whether or not the same mechanisms that kept us from annihilating the planet would hold there too.

I am not sure that it will.

But a balance duopoly tends to be better than a monopoly and if there are multiple poles then so much the better.

I'm past the point where I believe that any of these a-holes is in it 'for the good of humanity', their behavior so far can be extrapolated safely to what they will be like when they are in the control of what effectively would be the largest private arsenal ever created.

For a while I thought the AI doomers were off their meds and too paranoid but I'm coming around to the fact that they may have a point: we are rapidly approaching a point of no return, if we are not already past it and the people in charge are exactly the wrong people to be in charge at such a point in time.

How specifically would a balanced duopoly be safer?

> If the position is that one of these models will go out of control and destroy humanity then why would having two such models be safer than one?

Models don't go out of control, they generate totally harmless tokens. The people directing the models go out of control. The owners of the single powerful model are seriously tempted to go rouge and they're going to do it, sooner or later.

Two or more competing groups force the development of defensive capabilities which require strong models too. In addition, offensive capabilities in possession of different competing groups introduce the option of MAD (Mutually Assured Destruction) which is a form of defense that doesn't require defensive capabilities. Worked with nukes.

And what framework would there be for international cooperation? Without that element, this would just be self-destruction in a global race.

Maybe we're about to test the scifi trope that a common enemy - like a superior invading alien life form - is the only thing that could unite all of humanity.

They will if it would be against international treaties to do otherwise, mutually enforced by every other world power.

We don’t even need something that powerful, you can control the hardware manufacturers and distributors. If you control GPUs, HBM, and the hardware needed to cool all that down and provide the power, that’s enough to make it very hard to scale up the risk.

You cannot really hide an AI datacenter consuming GW of power for too long.

Except no one country controls all that. And if you put limits on it, other countries will speed up doing it themselves.

If you stop going "nobody will do anything actually effective", and start going "how do we do something actually successful", any number of successful ideas start readily arising. It is possible to stop. Not easy, but possible.

Like a war or something? I can't think of a successful non-bad way for everyone to stop besides a treaty.

Outside of the current nonsense in Iran, we've never had to go to war to prevent nuclear arms. We've never had to go to war to ban weapons in space. These things are possible. Hard but possible

> we've never had to go to war to prevent nuclear arms. We've never had to go to war to ban weapons in space

We've mostly been blindsided by covert nuclear programmes. As for weapons in space, they don't have any purpose. The moment they make sense we'll see weapons in space.

That's why treaties sound reasonable to me. Though I'd imagine they would be more like Climate Change ones. Before Iran there were bombings to stop nuclear arms. Imagine a country getting bombed because it's detected their building a data center.

US + South Korea + Taiwan gives a control over pretty much all the hardware used for AI training and inference. Of course having a psychopath president like Trump doesn’t really help at anything that requires international collaboration

They made a movie about that.

Between:

   "Iron Sky"
   "Whoops Apocalypse" 
   "Wag The Dog"

and

   "Idiocracy"

You'd think we were amply warned.

> international treaties to do otherwise, mutually enforced by every other world power

This does not exist.

Look at how we regulate nuclear weaponry. Consider how much more widespread it would be if we hadn't.

Nuclear weaponry is effectively regulated because the nuclear powers have agreed that nobody else should be allowed to have the capability.

So this is basically suggesting that USA and China should work together to ensure that they are the only ones allowed to develop and maintain capable AI. And between the two of them they can carve up and techno-colonize the rest of the world.

> Nuclear weaponry is effectively regulated because the nuclear powers have agreed that nobody else should be allowed to have the capability

Nuclear powers also gave security guarantees that e.g. convinced Ukraine to give up its Soviet stockpile. We're almost certainly entering a new regime of proliferation due to the breakdown of that system.

We've also agreed, for instance, not to do certain types of testing anymore. Just as we could agree to stop training even larger frontier models.

Except...

(From WP)

India

In 1998, India said it would only sign the treaty if the United States presented a schedule for eliminating its nuclear stockpile, a condition the United States rejected.[18]

Israel

In 2016, Israeli Prime Minister Benjamin Netanyahu said that its ratification was dependent upon "the regional context and the appropriate timing".[19]

United States

The United States has signed the CTBT, but not ratified it; there is ongoing debate whether to ratify the CTBT.

https://en.wikipedia.org/wiki/List_of_parties_to_the_Compreh...

I don't understand. They're slowing the development because they want to or because they have to? Because of ethical concerns? Or because of economics factors that prevent them from delivering results?

Since when have ethical concerns slowed the pace of technology?

I suspect it's all about the last bullet point:

> OpenAI said on Wednesday that it was pushing for mandatory national AI safety requirements in the United States, citing concerns that advanced AI systems could accelerate its own development.

They can just have codex develop part of its next version and oh look, it’s accelerating itself! So scary. </s>

Because it serves as a justification for regulation and government funding.

No, Altman is saying this exactly because he wants to avoid regulation.

This is a common tactic in any industry. Say that you will self-regulate, but then a decade or so later everybody wonders why anyone thought it was a good idea to allow an industry to make their own rules.

he wants to write the regulation.

> hey're slowing the development

They're doing no such thing. Altman is saying words ahead of the midterms.

My guess is: progress is stagnating, so they look best if they pretend to slow down intentionally while continuing to work at full speed.

My money is on this as well. Their model improvements are getting more marginal over time, they have no moat, Anthropic is moving past them and apart from software engineering and a few tech fields LLMs fail to deliver business value.

So the “slowing down” narrative is a convenient story to spin.

They don't deliver business value in software engineering either. It gives the _illusion_ of increased productivity, but a machine that drives your engineers to burnout in exchange for more code shipped that's still bottlenecked at all the other stages of software development is not a winning strategy.

The entire AI market is propped up on the hope that the models will magically solve all of these problems, and now that even Altman is realising they won't he needs the slowdown to stave of the crash for as long as he can.

Yes that and and Math problems don't pay the rent.

Yes. They will reach an agreement not to pursue RSI for “safety.” That RSI is utter nonsense in the first place will go unnoticed.

> In August, OpenAI paused much of its model development for two weeks to bolster its defenses after its AI agents escaped containment and hacked open-source platform Hugging Face.

That’s one sneaky way to remove OpenAI responsibility…

How does that imply anything like that?

Meta in their lawsuit negotiated 5B to be paid out only if other market players agree on implementing safeguards. In the same vein OpenAI can slow down competition, lock out opensource AI.

> OpenAI said on Wednesday that it was pushing for mandatory national AI safety requirements in the United States, citing concerns that advanced AI systems could accelerate its own development.

As well, the weaponisation of AI is already there, open ai can switch off safeguards at will. A massive swarm was allocated to solving Millenium problems in a few days. Imagine the US having such a weapon, they want to ensure other nations cannot have this kind of access. I imagine the US might be very interested in stifling globaal Innovation and be sympathetic to OpenAI for these reasons.

Timing is pretty good for OpenAI right now, given GPT-6 Astra is out and appears to match or even beat Claude Fable 5.1.

They might not be as interested in slowing down if Anthropic currently had a significantly better model than them!

I’m curious why you use this language of “appears to match or even beat”, seemed pretty clear from the benchmarks that it’s well ahead of Fable. Do you doubt the reporting?

I'm using both extensively at the moment and I've not yet found that either is demonstrably better at specific tasks than the other. They're both beasts!

(I've not had Fable do as much work in Blender as Astra yet though, and it's quite possible that Astra has Fable beat on 3D modeling.)

Maybe I can help with the curiosity - his own experience is probably that astra is close to fable, not miles ahead of it. Everyone has an opinion, and some people don't trust the benchmaxxed stats. Some of us just try it for our use cases and figure out which one we like best!

Fable still better for planning, Astra faster overall but misses things. Maybe that’s behaviour not raw intelligence but it’s noticeable

I doubt the reporting: I've found it to suck for productive work. Gone back to using Sol.

You know… I agree. I haven’t found it to be as great as reported.

All they're saying to the other players is: let's lock it down and slice up the market between us.

Claiming Skynet is just around the corner is a great way to get the public to believe only these huge companies can harness AI, and how all open-source models should be outlawed.

There are no open source models, at least not useful ones. Open weight is not the same as open source. Open weight models are still just inscrutable binary blobs that you can (theoretically) run on your own computer instead of through a SAAS web app. The open weight model labs don't even provide a high level catalog or any description whatsoever about what went into the training data.

This is not open source and we really should stop conflating the two things.

Is he trying to find a cop-out for his diminishing returns?

Absolutely this. Flip the idea on the head to reveal where he’s at coming out with such statements.

translation: we are burning cash faster than we're getting better, so lets frame the slowdown as intentional for greater good.

Words are cheap, and Altman is a salesman. He’s just staying whatever he needs to say to keep people off his back. I don’t put any stock into this statement whatsoever. He may slow development, or he may not, but whether he does will depend almost entirely on what’s best for him and his company, not humanity.

Maybe he will even give everyone a bonus, but only if they agree with his philosophy during the next employee survey. It’s a new fad I heard about.

Reminder that Altman does nothing but lie so this means nothing.

Of course they are - anything to secure their position.

Fable and Sol were pretty big step functions, but I wonder how much if this, from openai and anthropic, is knowing they are somewhat tapped out. That they don't have a ton more to give. Maintain the mystique under a veneer of, "oh we would build the magic wish machine if we were allowed, but we're not, here's Fable 5.8 instead".

Astra has a lot of new capabilities but man, Armin once again being an early lead indicator, and in this case showing that the model seriously has some screws loose. https://lucumr.pocoo.org/2026/9/7/astra-why/ https://news.ycombinator.com/item?id=49654229

do you think they will slow it down for the government too

If they had a sufficiently superior AI, they would not need an IPO, and they would certainly not be renting it to people. So look out for an end to new AI releases.