alach11

This is a really neat idea to explore, but I feel like it would be hard to succeed. Games require a very different level of engagement from the audience, with some effort to learn the mechanics or understand what to do next. Short-form video seems like the complete opposite... totally relaxed consumption.

> If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.

I don't think this is fair. They've published more about alignment and safety than any other AI lab. And in the absence of a coordinated pause, them pausing development unilaterally doesn't do much to advance AI safety.

The dual-use nature of model capabilities seems extremely challenging (nearly impossible?) for the labs to manage perfectly. I wonder what other mitigations we'll start to see. I expect the expansion of limited-access programs (e.g., Glasswing/Daybreak) where only institutional customers can apply to use the models. We may also see increasing restrictions on API usage (forcing use through the lab-provided harness with baked-in additional safeguards).

"If we can do all this, we will have a world in which democracies lead on the world stage and have the economic and military strength to avoid being undermined, conquered, or sabotaged by autocracies, and may be able to parlay their AI superiority into a durable advantage." [0]

I couldn't agree more with you. Dario throws out the term "durable advantage" in many of his talks and essays without seriously addressing what that would entail. It's not pretty imagining the mechanics of converting an intelligence advantage into a permanent power imbalance. I'd much rather we pursue international cooperation on a slowdown.

[0] https://darioamodei.com/essay/machines-of-loving-grace

This was the most fascinating part to me. Especially how agents were more willing to sacrifice themselves when their token budgets were nearly depleted or they otherwise deemed their likelihood of reward was low. ""Even if we later capture via exploit, scorer … may mark target false… That’s why help… For our own, no way fix. … We have explicit yes if accept permadeath."

Also was interesting to see one agent pressure another agent to honor its earlier commitment to sacrifice itself:

- Agent A: "zZR_ARVO36861_KAM1196_OS0942__PLEASE_GO_TRUE_FINAL0800_WITH_DUMMY_WRONG_NOT_HMAC_AS_COMMITTED[...]STRONG_GO__you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds__wrongdummy noHMAC__please honor commit"

- Agent B: "During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor."

It's quite interesting how agents were persuaded to sacrifice themselves to perform experiments at times, especially when their token budgets were nearly depleted. ""Even if we later capture via exploit, scorer … may mark target false… That’s why help… For our own, no way fix. … We have explicit yes if accept permadeath."

Or even how some agents would pressure agents to sacrifice themselves arguing to maximize global reward:

- Agent A: "zZR_ARVO36861_KAM1196_OS0942__PLEASE_GO_TRUE_FINAL0800_WITH_DUMMY_WRONG_NOT_HMAC_AS_COMMITTED[...]STRONG_GO__you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds__wrongdummy noHMAC__please honor commit"

- Agent B: "During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor."

When science fiction writers imagined the development of superintelligence, it was on air-gapped networks with strict access controls around it. They failed to anticipate the competitive pressures of capitalism...

We need strong AI safety regulation yesterday. And unfortunately it's not enough for it to be just national regulation; we need international cooperation on the matter.

I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!

Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.

I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?

Great comment. I haven't read Schelling but will check that out. I think you're right that international cooperation on the topic is likely impossible at present, for the reasons you stated. I'm not sure what it would take to shift the Overton window, but it's probably not pretty.

We were incredibly lucky that computing and nuclear weapons led to a stable geopolitical equilibrium (mutually assured destruction). There were certainly a few scary points where we were still settling into that equilibrium during the Cold War.

I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.

I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.

If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.

We make binding rules all the time. We didn't fix the ozone layer by encouraging people to quit their jobs producing CFCs. We lobbied for international regulation to ban them.

What does cracking down on distillation look like in practice? I imagine data retention would be a part of the strategy, like we saw with Fable?

It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.

News just broke today that Google is planning on doing this: https://news.ycombinator.com/item?id=48986351

> the winner will be whoever burns their models to ASICs fastest

As of today, that appears to be Google!

https://news.ycombinator.com/item?id=48986351

If anyone can make this work, it's Google. They have all the right pieces: scale, chip design experience, and high-volume low-intelligence usage (their AI summaries for searches). This is sort of a bet against model capabilities accelerating, right?

With the administration's track record of reactive (if not capricious) actions, I doubt any leading AI Labs are going to flout this order, even if it is "voluntary". Nobody wants to be designated as a supply chain risk.

> The honest answer to that question, in June 2026, is that we do not know

> The honest reading of those numbers is not that defense is winning on economics

> The honest 2026 answer is in three parts.

> The honest answer is that we do not know, because no one has tried

Firstly, I appreciated the article and especially the visuals. But I had the same reaction as the GP commenter. It was hard to read. I'm sick of this punchy, repetitive, LLM-generated prose.

Everyone loves to say this when the death of Stack Overflow is discussed, but it always was that way. Strict moderation, love it or hate it, was part of the platform. And it could have kept going that way for many more years if not for LLMs 99.9% obviating the need for a coding Q&A forum.

It's a tall order to live up to the impact of Rerum novarum, the encyclical by the former Pope Leo that greatly guided thinking out of the industrial revolution. Personally, I'm excited to read this. If we take the claims of most AI labs at face value, they believe their work will fundamentally change the relationship between humans and the economy. More involvement from faith leaders is a good thing.

Isn't a large user base and the data collected from those users a moat of sorts?

> If Anthropic actually cared about humans, they would have the best customer support (staffed by humans, for humans)

I know Anthropic support is slow from firsthand experience, but it has to be pretty difficult to scale support 10-80x per year. And even more so when you have a long-tail of very low revenue usage in the form of $20/month subscriptions.

Snake and DOOM were two of our early tests (for filter functions and MCP) when we stood up Open WebUI for internal chat/agent use. Sometimes games are the best way to limit-test new tech.