Do you have a source on METR employees retaining massive equity stakes?
This conspiracy theory is truly crazy. A $20K scholarship in 2022 is supposed to explain Coxon walking away from unvested equity for a company worth over $950 billion dollars?
I don't think Coxon was ever planning on or entitled to taking equity, I think this was the plan from the beginning and why he was hired for 6 weeks to begin with.
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
And how does that relate to the ask for oligopoly licensing within global democracy?
“We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
All “tech workers” are.
I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again?
> And how does that relate to the ask for oligopoly licensing within global democracy?
They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism.
> “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant.
Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
Selection effect.
Everyone who thinks "the biggest difference I can make is staying in/joining/founding new AI research lab" does that.
Everyone who thinks "the biggest difference I can make is leaving/whistleblowing", does that.
Treating both groups as the same by virtue of employer is the goomba fallacy.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry.
The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies with obscure money trails. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice.
And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
They're not proposing anything concrete, and when they do, what do you think the proposal will be? Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies? Or are their proposals going to be aimed at everyone running open Chinese models?
> Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies?
This is compatible with the language being used, but I suspect they're not going to do that.
And if a threat to the human race does come from AI, it's going to come from OpenAI/Anthropic. Hypercapitalist, secretive, in bed with the government, plus multiple real documented hackings of open source infrastructure already.
We're a hell of a lot safer with China doing the same research out in the open and making it available to anyone. The choice might well be: one or two superintelligent autonomous AIs at OpenAI/Anthropic - or a lot of smaller ones, unable to be controlled but also coming out of a diverse set of environments.
One of those leads to a stable ecosystem where we can all coexist, the other is genuinely terrifying. But make no mistake, from OpenAI/Anthropic this is all motivated by their stock price - when you're in the silicon valley mindset, it distorts your reality. They've convinced themselves that everyone's safer if they stay on top and in control, conveniently ignoring how that benefits them, and I don't believe them for a minute.
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from denial as you can get).
Threats. Like they claim to be.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively?
If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
Megalomania is not evidence of either a problem or a solution.
Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this.
God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture.
This is exactly the opposite of what Dario proposes.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
HAH
In fact they still have not caught up with February's Mythos, indicating they are more than half a year behind.
Mostly I heard this from people who I got the impression have little experience in developing greenfield software with agentic AI. Often the same people who talk about spec frameworks.
I fundamentally disagree with the approach. I believe the ability to autonomously evaluate, test, and adjust during long horizon tasks is critical to using AI efficiently.
I also disagree with the approach, this is cargo-culting the existing ways of working.
I think you mean the existing anti-patterns. They don’t call them ivory tower architects for nothing.
they are more efficient and require less compute for the same thing.
they are ahead in specialized tasks.
they are ahead in areas closed models refuse to answer.
they are ahead in emotional intelligence.
For speed and efficiency, you are most likely wrong.
Speed is led by GPT-5.6 Sol on Cerebras Ultrafast at 750 t/s. Afaik you cannot serve a single DeepSeek Flash 4.1 stream at 750 t/s, plus the model is less intelligent as seen on newer benchmarks.
I believe OpenAI and Anhropic are at the frontier of efficiency too. There were numerous reports about their breakthroughs and associated API price cuts. The idea that open-weight models are more efficient seems unfounded.
5.6 sol ultrafast on cerebras is 750tps, open models readily exceed this. just by using a smaller model cerebras serves qwen 3.8 27b at 1850tps. or even larger models, mimo 2.5 pro was served for a while at 1000tps. and so on. [https://inference-docs.cerebras.ai/models/choose-a-model]
the chinese ai companies have 10% of the total compute resources of the US ones. since the USA tries to stop them from buying nvidia gpus. they maxed out the efficiency.
deepseek v4.1 has engram architecture. it has 550b params instead of 5T+ for astra/fable. it has 8b active instead of potentially hundreds active for astra/fable.
compare input/output/cache: $0.15/$0.60/$0.003 for v4.1 to $10.00/$50.00/$1.00 for astra and $10.00/$50.00/$0.25 for fable.
astra cache reads are over 330 times more expensive.
at the artificial analysis 7:2:1 ratio, deepseek is $0.18/m, fable is $7.18/m, astra is $7.7/m.
but what about intelligence? AA would rate deepseek v4.1 at AA 40, astra is AA 53.
so it cost 4,180% more for 32% more intelligence.
they are serving that at over 250tps at baseten. to get close to that on astra API you are paying double the cost for fast mode.
so it is now 8456% more expensive for a similar speed and 32% more intelligence. 84 times more expensive.
you could compare gpt 5.6 luna. if you did that the same way as before you would get a blended price of $0.17 for luna at AA 38. for baseten it would be $0.20 for v4.1 at AA 40.
assume roughly the same intelligence. on AA openai gets 117tps. baseten gets 284tps. so 18% more expensive but 142% more tps.
the fast mode is again double the cost, roughly same intelligence. so luna in that case would be 70% expensive. take the per task cost and it would still 9% more expensive.
so i think there is something to be said about the efficiency of the model.
https://artificialanalysis.ai/ intelligence vs cost per task disagrees with this statement.
2. For public-facing models, the differences are really minor with some occasional model (like Fable or Astra) showing some better performance in specific benchmarks for the span of some weeks or few months before open ones catch it.
3. Being bleeding edge is overblown anyway in the real world, besides the occasional "very latest fresh model did this task which previous one couldn't", and the number of those tasks is increasingly small and far from mundane corporate needs.
This is not sustainable anyway, the scale at which they want to control data flows. Time was money, now data is money and they are too greedy for it.
All this agitation is just a silly attempt at constraining competition. The danger is not the AI, it is having all your systems connected. Overreliance on networked tech.
If correct, grip is tightening. Used to be 1y, then 6 months ...
While the data pipeline is necessary, the assumed source for data (in this case people) is incorrect. Right now models are "aligned" because of the RL-based people pipeline.
But there's a whole world of readily available data that does not require a human to access. Put sensors on a vacuum, install lidar on the front of a car, sell phones with cameras on them, and you amass data for the creation of models of the world that don't include the need for a human filter. Richard Sutton and many others see this approach as the only viable path to AGI. Such models would be truly alien and unaccessibly dangerous.
We're probably building them right now.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.