News for hackers. Stuff that matters.
Re:No, it doesn't cost Anthropic $5k per Claude Code user by simianwords on Tuesday March 10, 2026 @02:45AM (#47319813)
>It is not. It's a terrible comparison. Qwen, deepseek and other Chinese models are known for their 10x or even better efficiency compared to Anthropic's.

I find it a good comparison because it is a good baseline since we have zero insider knowledge of Anthropic. They give me an idea that a certain size of a model has a certain cost associated.

I don't buy the 10x efficiency thing: they are just lagging behind the performance of current SOTA models. They perform much worse than the current models while also costing much less - exactly what I would expect. Current Qwen models perform as good as Sonnet 3 I think. 2 years later when Chinese models catchup with enough distillation attacks, they would be as good as Sonnet 4.6 and still be profitable.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by coldtea on Tuesday March 10, 2026 @07:16AM (#47321711)
> I don't buy the 10x efficiency thing: they are just lagging behind the performance of current SOTA models. They perform much worse than the current models while also costing much less - exactly what I would expect.

Define "much worse".

  +--------------------------------------+-------------+-----------+------------------+
  | Benchmark                            | Claude Opus | DeepSeek  | DeepSeek vs Opus |
  +--------------------------------------+-------------+-----------+------------------+
  | SWE-Bench Verified (coding)          | 80.9%       | 73.1%     | ~90%                 |
  | MMLU (knowledge)                     | ~91         | ~88.5     | ~97%               |
  | GPQA (hard science reasoning)        | ~79–80      | ~75–76    | ~95%             |
  | MATH-500 (math reasoning)            | ~78         | ~90       | ~115%            |
  +--------------------------------------+-------------+-----------+------------------+
Re:No, it doesn't cost Anthropic $5k per Claude Code user by yorwba on Tuesday March 10, 2026 @08:01AM (#47322070)
Where are you getting those benchmark figures from? Math-500 should be closer to 98% for both models: https://artificialanalysis.ai/evaluations/math-500?models=de...
Re:No, it doesn't cost Anthropic $5k per Claude Code user by Filligree on Tuesday March 10, 2026 @07:29AM (#47321809)
Everyone who's used Opus knows it's better than the others in a way that isn't captured by the benchmarks. I would describe it as taste.

Lots of models get really close on benchmarks, but benchmarks only tell us how good they are at solving a defined problem. Opus is far better at solving ill-defined ones.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by coldtea on Tuesday March 10, 2026 @07:36AM (#47321874)
>Everyone who's used Opus knows it's better than the others in a way that isn't captured by the benchmarks. I would describe it as taste.

Ah, the "trust me bro" advantage. Couldn't it just be brand identity and familiarity?

Re:No, it doesn't cost Anthropic $5k per Claude Code user by vidarh on Tuesday March 10, 2026 @08:18AM (#47322203)
I have a project where we've had Opus, Sonnet, Deepseek, Kimi, Qwen create and execute an aggregate total of about 350 plans so far, and the quality difference as measured in plans where the agent failed to complete the tasks on the first run is high enough that it comes out several times higher than Anthropics subscription prices, but probably cheaper than the API prices once we have improved the harness further - at present the challenge is that too much human intervention for the cheaper models drives up the cost.

My dashboard goes from all green to 50/50 green/red for our agents whenever I switch from Claude to one of the cheaper agents... This is after investing a substantial amount of effort in "dumbing down" the prompts - e.g. adding a lot of extra wording to convince the dumber models to actually follow instructions - that is not necessary for Sonnet or Opus.

I buy the benchmarks. The problem is that a 10% difference in the benchmarks makes the difference between barely usable and something that can consistently deliver working code unilaterally and require few review interventions. Basically, the starting point for "usable" on these benchmarks is already very far up the scale for a lot of tasks.

I do strongly believe the moat is narrow - With 4.6 I switched from defaulting to Opus to defaulting to Sonnet for most tasks. I can fully see myself moving substantial workloads to a future iteration of Kimi, Qwen or Deepseek in 6-12 months once they actually start approaching Sonnet 4.5 level. But for my use at least, currently, they're at best competing with Athropics 3.x models in terms of real-world ability.

That said, even now, I think if we were stuck with current models for 12 months, we might well also be able to build our way around this and get to a point where Deepseek and Kimi would be cheaper than Sonnet.

Eventually we'll converge on good enough harnesses to get away with cheaper models for most uses, and the remaining appeal for the frontier models will be complex planning and actual hard work.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by oren1531 on Tuesday March 10, 2026 @09:23AM (#47322938)
Good point on the green/red dashboard. The opportunity cost angle is worth adding though. A failed run isn't just the wasted tokens and retry cost - it's also the task that didn't get done and the engineering required to diagnose why. On anything time-sensitive, that compounds fast.
Re:No, it doesn't cost Anthropic $5k per Claude Code user by vidarh on Tuesday March 10, 2026 @10:40AM (#47323940)
Exactly. At the moment it's close enough to be a wash for some cases, or tilts seriously one direction or other for others. I expect improved harnesses means more and more we'll just be able to re-run a couple of times, and fall back to "escalating" to Sonnet or even Opus, but whenever it involves egineering time, that's a big deal.
Re:No, it doesn't cost Anthropic $5k per Claude Code user by Bombthecat on Tuesday March 10, 2026 @09:08AM (#47322788)
In 12 months, opus will be better than now and you still won't use it lol
Re:No, it doesn't cost Anthropic $5k per Claude Code user by vidarh on Tuesday March 10, 2026 @10:38AM (#47323904)
I still won't use what? I use Opus now, and I will use Opus then too, but as I clearly stated:

My default model has now dropped to Sonnet, because Sonnet can now do most of my tasks, and we already use Kimi, Deepseek, and Qwen.

They're just not cost-effective enough to be my main driver yet. They are however cheap enough that for things where the Claude TOS does not let me use my subscription, they still add substantial value. Just not nearly as much as I'd like.

The bulk of my tasks won't get harder as time passes, and so will move down the value chain as the cheaper models get better.

For the small proportion of my tasks that benefits from a smarter model, I will use the smartest model I can afford.

[dead]
Re:No, it doesn't cost Anthropic $5k per Claude Code user by vidarh on Tuesday March 10, 2026 @12:54PM (#47325827)
Thankfully it's not as bad as that. The 50% that goes red means we re-execute those steps, potentially several times, to see if they succeed, before we even bother manually looking at it. But the overall principle holds: First yoou multiply the cost by re-running, then eventually you either need to kick it up to a more expensive model and/or a human.

But of course this is also only viable for non-latency sensitive work, for starters.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by cesarvarela on Tuesday March 10, 2026 @09:48AM (#47323214)
The harness makes a difference too.
Re:No, it doesn't cost Anthropic $5k per Claude Code user by ACCount37 on Tuesday March 10, 2026 @09:44AM (#47323157)
One of the main edges Anthropic has is that "personality tuning" gap. "Nice to use" is a differentiator when raw performance isn't.

OpenAI can sometimes get an edge over Anthropic in hard narrow STEM tasks. I trust benchmarks over vibes there - and the benchmarks show the teams trading blows release after release. Tracking Claude Code vs OpenAI Codex on SWE-bench Verified feels like watching the back alley knife fight of the AI frontier.

But the vibe of "how easy is that model to interact with" and "how easy it is to get it to do what you want it to" does matter a lot when you are the one doing the interacting. And Opus makes for a damn good daily driver.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by cmrdporcupine on Tuesday March 10, 2026 @10:27AM (#47323766)
At this point it's frankly not a fair comparison since DeepSeek 3.2 is now many months old and we're waiting for a newer model which has been rumoured as "any day now" since February. (We'll see).

GLM5, the largest Qwen 3.5 model, and Kimi K2.5 are more fair comparisons, though they are, yes, a bit behind. They're more than capable for routine operations though.

Anyways, I'm back to using Opus & Claude Code after a month on Codex/GPT5.3 and 5.4 and it's frankly a rather obvious downgrade. Anthropic is behind OpenAI at this point on coding models, and there's nothing to say they couldn't fall behind the Chinese models as well.

The moat is very shallow. After the events of the last two weeks there's likely a significant % of international capital very interested in breaching it. I know I would like to see this... Anthropic basically said F U to any non-Americans, and OpenAI is ... yeah.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by torginus on Tuesday March 10, 2026 @11:55AM (#47324986)
Dunno, I was using Cursor today and for some reason it decided to swith to GPT 5.3 at some point and I didn't even notice. I was sure that Opus is much better, but who knows?
Re:No, it doesn't cost Anthropic $5k per Claude Code user by crooked-v on Tuesday March 10, 2026 @12:17PM (#47325275)
> distillation "attacks"

I find it really funny that anyone can call it this with a straight face when all the American models are based on heaps of illegally pirated books and TOS-breaking website scraping in the first place.

Re:No, it doesn't cost Anthropic $5k per Claude Code user by hirako2000 on Wednesday March 11, 2026 @03:14AM (#47332514)
Looters calling the police for being looted.

Plus, Chines made distillation did good to the overall internet infrastructure. Millions of small Joe's WordPress website, paid and maintained out of good will, getting hammered by AI mining vs making the VC loaded pirates pay for what they seeded, I find the latter more fair.