mlinsey

That market resolves using the arena.ai text leaderboard, where Argon was already #1 over Opus 5.5. Coding is way more economically consequential, and I think the current gap between Anthropic and Google is very wide there (at least, before this Carbon release)

I understand that the Exponential Time Hypothesis being false doesn't necessarily mean P=NP, but for people who are a lot more knowledgeable about complexity theory, does this update your priors significantly in the direction of P=NP?

For coding use cases, it really isn't.

DS4.1 flash is $0.30 in / $1.20 out (per M, peak, cache miss) Opus 5.5 is $4.00 in / $20 out (per M, cache miss)

However, that is API prices.

Anthropic offers a $200/mo subscription. How this translates into usage is admittedly a bit opaque, subject to change, and depends on how exactly you use it. But it's a lot of usage - Semianalysis data shows that $200 is getting you around $2,500 of usage at API rates if you use Opus 5.5. This is close to what I'm seeing anecdotally with my accounts, if anything I have been getting a bit more.

Now, unlike DeepSeek, you can't use your subscriptions to power live AI-driven products, or resell tokens in any way. But for personal coding agents, you can use as many of these subscriptions as you want, for now. So I am paying effectively basically 8% of the published API rates, so at my usage:

DSv4.1: $0.30 in/ $1.20 out Opus 5.5: $0.32 in / $1.60 out

Obviously, those aren't real prices, but they accurately convey apples to apples what my everyday usage costs me and most other heavy users, and why it's so easy for me to stick with Anthropic/OpenAI.

I don't think it's a coincidence, either - I think the token allowances for these subscriptions are set to be competitive with the open models, so that most coding users (and their incredibly valuable data) stay with the frontier labs, while VC-funded wrapper companies and less-price-sensitive giant companies with strict procurement policies pay exorbitant markups for enterprise contracts at the API rate.

I'm paying for the heavily-discounted subscriptions, not the API rates. There isn't really a cost gap for me. DeepSeek doesn't have a subscription to compare to, but when I compared the GLM 5.3 usage I got from a $100/mo Z.ai subscription compared to Opus 5.5 on a $100/mo Claude subscription, there wasn't a big gap. And GLM 5.3 is very clearly not a frontier model (deepseek v4 seemed a lot closer, but I didn't use it enough to really say for my workloads).

I don't think those subscriptions nave negative contribution margins, either. I think we're seeing a lot of price discrimination by the big labs, and huge margins on their frontier models. The fact that they have been cutting prices to their second-biggest tier of models (Opus/Sol).

Open models catching up and collapsing these margins would worry me if I were a shareholder in the big labs, but as a user, I really doubt that the western labs have bigger environmental impact just because they have higher API costs, I think they have a ton of efficiencies they aren't sharing with customers yet because demand is so high.

Not to get back into the AI safety debate from last week, but I really, really want a team of human experts to understand what the AI is suggesting before we administer new medical treatments or synthesize new lifeforms in the lab.

Especially with these big models chewing up limits fast, I feel good about simultaneously having an Anthropic sub, an OpenAI sub, and an Opencode balance. The models also seem to catch things when code reviewing each other that they don't always catch when a new instance of the same model does a review.

Revoked or not, just ever having those controls signals to the Chinese ecosystem that you're not necessarily a reliable supplier (Would you trust US export policy to remain stable for the next ~decade given the state of US politic?) and to the Chinese government just how strategically important you see these components.

This isn't the kind of thing you can hash out in public and go back and forth on. Once you put it out there, the other party will take steps to make sure they don't have to rely on us in the long run.

High-performing open weight models being released recently, and your customers looking into working with multiple providers as a result, are a great reason to drop prices on your non-frontier offerings.

Although I'm sure there are some efficiency gains, the technology is too new and labs are scrambling to release too quickly to think that the low-hanging optimization fruit has been picked already.

"With one big caveat: you need a very clear set of requirements"

Most new product launches are an exercise in figuring out what the product requirements should be through trial and error (really: through ongoing dialogue with your users). Even mature products can have requirements change over time as the market changes.

I think internalizing this reality is why most senior engineers who work in domains that touch the messy real world will reflexively push back against perfectionism.

For ordinary coding (making webapps, frontend and backend), the non-SOTA models do fine, but the SOTA models have better judgement and need me to intervene less often, allowing workflows like "Go and pitch me a solution to this problem, then go build it". 6 months ago, I would give lower-level tasks and read more of the code myself.

So I don't "need" the SOTA models, but they do help considerably. And I am also working on some other projects that may have been outside the reach of the earlier models entirely.

I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.

This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.

I think he's anticipating and discarding this objection with his introduction, which otherwise feels disconnected from the rest of the article. FWIW, I have changed a bike tire and I'm pretty sure most of the MTS at the big labs could. This sort of "they're just bookworms who don't understand the physical world" rhetoric aside, we are currently seeing a ton of effort and expense go towards giving the AI agents hooks into being able to perform as many real-world-consequential actions as possible. And you can do a surprising amount with just bits, from writing code to breaking into systems to sending some combination of emails, phone calls, and currency to instruct meatspace humans to do things, etc.

Also, I just want to step back and actually give Costco some props here for their brand loyalty. Inshallah, when I have a 400 Billion market cap company and someone writes a fawningblog post about how awesome I am from a social-benefit perspective and how socialist mayors should learn from me…it’ll be the skeptical comments that will be suspected as paid shills for my $2T competitor!

You got me, I'm totally paid off. Real New Yorkers love to go five miles (just for the closest one to me, in Manhattan seven avenues over and around 100 streets north), take a cab back from shopping, then haul costco-sized boxes up the stairs. I only pretended to not love that to get the sweet check from daddy Bezos (good thing he didn't notice my aside criticizing the e-bike trailers breaking traffic laws, though I guess that was subtle if you haven't been following the controversy about them here)

Costco is an elegant solution for the suburbs, where everyone is driving around a vehicle large enough to store giant boxes off a pallet and bring them home. Here in NYC, it's really impractical to go to a warehouse and carry a month's worth of supplies home on the subway. The flip side is that the Amazon last mile deliveries are done on electric scooters that can bring a whole trailer worth of packages to the mailroom of a big apartment building. These have some other externalities (eg around traffic laws and sidewalk space) but they are on the whole a lot more resource-efficient than everyone driving around giant cars to go to the store.

I'm not sure where in my post you thought I was suggesting either ignoring an export control or that all export controls are illegal.

Export controls are governed by the Export Control Reform Act of 2018. The procedures the government must follow to enact a rule is defined by the Administrative Procedures Act. It's not a slam-dunk case, but I think there are significant questions around the arbitrary nature of the approved company list, the control restricting US-based permanent resident employees from accessing models, and how the rule was enacted in general which would make a viable case.

Suing the government saying that APA was not correctly followed or that the export control in question does not comply with the ECRA will absolutely not get you put in prison, that's not the kind of country we live in and honestly I have no clue what you're talking about.

The labs will not just ignore the order, there are too many other levers they can try to pull to mess with those companies. Just for some examples, think about the number of employees reliant on visas that could be revoked, the government contracts that the hyperscalers hosting them that could be canceled, the certifications that all the data centers need to be hooked up to the grid, the tariffs that could be put on critical components, the IPOs that need to be approved by regulators, the bill introduced in Congress to seize 50% of their equity...

Lots of these moves would and should be struck down in court as an arbitrary and capricious use of administrative power. Some of them might not be, and in the meantime you're signing up for tons of trouble. A trillion-dollar company does not simply go to war with the US government.

A more mid-sized company that's not so intertwined, but not so small that they can't get a good legal team, might be another story.

In 2004, I took a class where we trained "language models" that were bigram word models, on an archive of a couple years of the Wall Street Journal.

I remember someone who literally announced they were dropping the class to the whole room at the end of a lecture, saying "This isn't AI!!!"

I understand why Anthropic might not want to fight this particular one in court, because they're trying to convince the administration to let them move forward.

But would another company who is not on the trusted partner list and has less to lose taking on the admin have standing to sue here? On the basis of the export control being illegal and this putting their business at a disadvantage vs. competitors with access

ID checks are possible for first-party harnesses...but they would also mean no more API access. Your wrapper could easily become a way for a foreign national to query Fable. Maybe a few large customers like Cursor would work with Anthropic to prove they had implemented ID checks themselves as well in their own products, but being able to just get an API key and have your product call frontier models may be over.

If the existing memory makers retains control of the market and don't defect from the optimal-long-term equilibrium for themselves, that's true. It just takes one player to defect for short term gains as we've seen with some past boom-and-bust cycles. Alternatively, it takes a sufficiently-resourced player with enough incentive to enter the market themselves (NVidia, Google, Amazon, the PRC government through one of many companies...)

You're describing efforts by powerful institutions to squash the technology, which they definitely try to do, but that's just a strong signal that the technology itself is inherently opposed to centralized power, not an enabler of it.

Other technologies like surveillance (and, perhaps, AI) are more clearly centralizing and enabling of power.

The difference matters a lot if you're having mixed feelings about working in technology.

That's definitely too broad a statement. I'd argue encryption, oral contraceptives, and the printing press were all strongly decentralizing.

My anecdata is that it heavily depends on how much of the relevant code and instructions it can fit in the context window.

A small app, or a task that touches one clear smaller subsection of a larger codebase, or a refactor that applies the same pattern independently to many different spots in a large codebase - the coding agents do extremely well, better than the median engineer I think.

Basically "do something really hard on this one section of code, whose contract of how it intereacts with other code is clear, documented, and respected" is an ideal case for these tools.

As soon as the codebase is large and there are gotchas, edge cases where one area of the code affects the other, or old requirements - things get treacherous. It will forget something was implemented somewhere else and write a duplicate version, it will hallucinate what the API shapes are, it will assume how a data field is used downstream based on its name and write something incorrect.

IMO you can still work around this and move net-faster, especially with good test coverage, but you certainly have to pay attention. Larger codebases also work better when you started them with CC from the beginning, because it's older code is more likely to actually work how it exepects/hallucinates.

The consumer surplus is quite high. Even with the regressions in this postmortem, performance was above the models last fall, when I was gladly paying for my subscription and thought it was net saving me time.

That said, there is now much better competition with Codex, so there's only so much rope they have now.

It's true he could write off xAI today and the company could still fetch a trillion-dollar valuation. But I was more referring to his stated intentions - between his stated plans, his actions taking SpaceX from a profitable company to spending basically all their revenue (plus a rumored large chunk of what's raised via its IPO) on AI, and seeing his tendency to make bet-the-farm bets on Tesla, I think it's fair to say he's committing to bet all of SpaceX on xAI.

Is this cash or compute? Elon has one of the world's biggest compute clusters spun up, and little inference demand to speak of.

Trading billions worth of idle compute, in exchange for a high-strike call option on the #3 player in the most-promising-vertical for AI, plus (presmuably) some access to their data, starts to sound like not a bad trade. Especially if you're pre-committed to betting your entire rocket company on winning in AI, and you're currently in sixth or seventh place.

Yes, cost per successful task is rising - ie, we are all paying effectively more for AI.

And yet - Anthropic is still struggling to have enough capacity to serve demand - they are virtually sold out.

And yes, are almost-as-good open models, on part with the closed models from 6 months ago (at worst), that are just a single Openrouter API call away, and yet Anthropic is still selling out. So people are paying for the premium product anyway, for whatever reason - maybe the last bit of intelligence is worth it, maybe they like the harnesses/products around the models, maybe it's a brand/enterprise sales thing.

Put aside your feelings about the AI industry and imagine we are talking about thingamajigs. Prices for thingamajigs are going up. They are still selling out about as fast (or faster) than the company selling them can build factories. There are more cost-effective competitors already in the market, but thingamajigs are selling out anyway.

Would you, looking at the thingamajig industry, conclude the "jig is almost up"? That "the returns aren’t anywhere close to what investors expect" and that the impending IPO is all some desperate hail mary to save things before the collapse?

You're observing that:

a) effective price-per-token is rising b) there is insufficient compute to meet the demand.

And your conclusion is that the industry is circling the drain and due to collapse?

The models that we are paying to generate tokens are already not really just LLMs, as anyone studying language models ten years ago (or someone who describes them as "next token predictors") would understand them. Doing a bunch of reinforcement learning so that a model performs better at ssh'ing into my server and debugging my app is already realllly stretching the definition of "language pattern".

I think when we do get AI that can perform as well as a human at functionally all tasks, they will be multi-paradigm systems; some components will not resemble anything in any commercial system today, but one component will be recognizably LLM-like, and act as an essential communication layer.

I agree, but also the model intelligence is quite spikey. There are areas of intelligence that I don't care at all about, except as proxies for general improvement (this includes knowledge based benchmarks like Humanity's Last Exam, as well as proving math theorems etc). There are other areas of intelligence where I would gladly pay more, even 10X more, if it meant meaningful improvements: tool use, instruction following, judgement/"common sense", learning from experience, taste, etc. Some of these are seeing some progress, others seem inherent to the current LLM+chain of thought reasoning paradigm.