Also, there a good technical reasons for inference being much more efficient at scale.
But that's not the point I'm making. (or, it kind of is, but it's more high level than that).
They're running spot and preemptible GPU instances (60-80% cheaper than on-demand), paying wholesale industrial electricity rates, and running at multi-tenant utilisation densities that make your MacBook look like a bonfire. Of course they're not individually loss-making on inference, they're aggregating cheap commodity compute and skimming a margin, and on paper that's what makes it seem like a good idea, certainly not a loss leader right?
But zoom out a bit; the entire stack is swimming in VC money. OpenRouter itself just raised at a $1.3B valuation backed by a16z. The Chinese models that now account for 36% of all tokens routed through the platform (DeepSeek, Qwen) are priced the way they are because Beijing-adjacent capital has decided market share matters more than margin right now.
So yes, technically no single party is "throwing money away" on each token; they're just all simultaneously subsidising different parts of the stack for strategic reasons. The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.
All that says is that it gets more expensive in the future as competitors exit the market and sustainability becomes important. That’s why Uber and Lyft were so cheap until they killed taxis. One major difference of course is that some models will remain largely good enough and the incremental cost of running will keep dropping to 0 over time since the hardware needed doesn’t get more expensive and is already purchased.
I only object to taking current prices as if they are perpetual prices.
I think the question in terms of throwing money away isn't the inference layer: it's whether the companies training open models will be able to financially keep doing so. How long will Moonshot keep releasing future Kimi models? I think there's an interesting wedge they're exploring with being basically a base-model-trainer-as-a-service, i.e. selling rights to Fireworks to sell finetuning services to the Cursors of the world, but it's entirely possible it doesn't pan out.
That being said, Nvidia seems willing to step up to being the base model trainer of last resort via the Nemotron family of open models, since it helps sell more of their hardware — similar to their investments in the CUDA stack to sell hardware (unsurprisingly, Nemotron is designed to run most efficiently on Nvidia hardware, e.g. native NVFP4). So I suspect there will continue to be a pretty good market here.
I'll invest in inference hardware whenever the economics make sense, not because of my prediction that prices will get higher (also, because smaller models keep getting better and they might just suffice for most of my use cases in a few years time).
My biggest worries in terms of cost is in regards to training. Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.
Valid takeaway.
"Invest" in the companies that release their models, because the alternative is to reward the people who will capture the market and jack up the prices to unreasonable level.
Unfortunately I think this is a bit like game theory, people will choose the local optimum despite the long term problems with doing so.
On Apple Silicon you can get 4x-8x more tokens per second if you run more queries in parallel (as long as your inference server supports it, and has enough spare RAM for more KV caches).
When inference is done at datacenter scales, when you distribute generation across multiple GPUs and have kernels carefully tuned to specific hardware, the compute vs DRAM bandwidth speed ratio gets absurd like 200:1. That's why everyone gives you batch inference at a steep discount.
Amazon were losing money, they were losing money because were growing and spent all of their cash flow on growth. It wasn't merely regarded as a hopelessly unprofitable business, if was regarded as potentially fraudulent. The share price collapsed in 2014 because, some thought, the profit would never come, investing in growth was pointless, etc.
Last year Amazon made nearly $100bn in profit. Stock is up 20x from then...this is after AWS was known (everyone also that was a massive fraud, could never be profitable...we know it was printing from day one), after it was the world's biggest retailer, etc.
It is difficult to understate how consistently people make this mistake, not just individually but in aggregate. You see the same thing with restaurants, consumer products, office leasing, so many businesses. This is not to say that the future will happen any particular way but that what Anthropic and co are doing is obviously rational and based upon very real cash flow. Anthropic's growth in revenue is, I believe, unparalleled in modern corporate history. A slight difference in this case is also that the economics of training these models is improving exponentially over time.
In this analogy, model training would be akin to developing better oranges, but your competitors are also developing better oranges so if you stop spending heavily to improve your oranges, consumers are going to buy ~zero oranges from you within a couple years. (Expanding the farm might be analogous to expanding data centers.)
The restaurant next to the mines were profitable up until the moment the mines themselves shut down: one doesn't exist without the other.
You can't ringfence inference as "the profitable bit" and then hand-wave away the training. Without continuous training there is no inference product.
Claude 3 Opus isn't sitting there making revenue in 2026 - the thing is just deprecated. The moment you stop spending billions on the next model, your "profitable" inference business is on borrowed time until someone else makes it obsolete.
Maybe I made a mistake in my analogy... They're not growing a farm and then selling oranges. They're on a treadmill where stopping is death, and the treadmill costs $10bn a year to keep running.
Unless they are changing the architecture in huge ways. The pre-training done for 3 goes into later models. I am sure the frontier labs are figuring out how to pretrain generic feedstocks that can be fed into downstream training pipelines. DeepSeeks incremental training run cost was what, 5M? Alibaba and DeepSeek have the best most efficient training pipelines, look at the rate at which custom Qwen models are being pumped out.
You’re literally describing all companies. Google takes about $270bn/year to run. If they stopped spending that they’d die pretty darn quick. It’s also a description of working - unless you’d built up significant savings, if you stopped working you’re also going to die.
No, not quite. It really comes down to opex vs capex and the depreciation schedule for your investment.
Software development is typically categorized as capex, on a 3-5 year depreciation schedule. You assume the software you write today will be generating value for you that long.
If a big, expensive model training project only gives you value for a year or less, that is not like most companies.
But believing that the financials of a project are governed solely by how IRS rules force you to account for headcount is kind of silly.
> If a big, expensive model training project only gives you value for a year or less, that is not like most companies.
The model itself that gets built? Sure (although clearly the timelines are getting longer). However the important bit here is the research that got done along the way and the infrastructure built to make that model building process cheaper, better etc. all of that stuff sticks around but because it’s hard to appreciate externally you discount it to 0 when it’s literally what they actually spent the money on.
But none of that even matters. Google had 270B in opex and their capex has grown from 50B in 2024 to 90B in 2025 and is projected to grow to ~175B for 2026. But even if you discount the “AI” treadmill, you’re still looking at many tens of billions in capex that if they stopped they’d die.
It's the same in other industries. In your example, if labor went into building a whole new assembly line that pays itself off over several years, capex. If it just goes into building products to sell, opex.
https://kpmg.com/kpmg-us/content/dam/kpmg/pdf/2023/tcja-chan...
There is no moat. In the end, what we are calling AI today will just be something that is incorporated into an existing programs that people will use to help them accomplish a task. The public will not be paying more for it. It will just be a commodity added to the existing ecosystems we have today. They
This claim deserves teasing apart.
Clearly, training is a Red Queen's race today. If a model provider were to unilaterally decide to stop training, they would very quickly lose market share to competitors with better models.
On the other hand, what if market and investment conditions change such that everybody has to stop training?
In that case, the models are still there and still as useful as they were the day before. So why wouldn't there still be an inference product?
The reason they're losing money on paper is because the models keep growing 10x in size every generation but they're not getting 10x returns on model inference (closer to 2x)
Dario's point was the opposite of yours; he used per-model accounting to explain why the company P&L gets worse every year, not better. His own numbers (10x training costs each generation and ~2x revenue return.)…
"It looks like it's getting worse and worse" are his words, not mine.
In 2025, Anthropic's inference costs came in 23% over their own projections. They cut their gross margin forecast from 50% to 40%.
The news they successfully buried was that companies like AirBnB are now running Qwen and open source models. The free oranges are now good enough. There is no future unless the goal is to get to super intelligence and utterly take over the world before anyone else gets one. Anything else and free models are six months behind. The money now is the opposite of what everyone thought a year ago: datacenters. Everyone thought AWS was fucked. Turns out AWS is really good at running Qwen.
I’m struggling to find the quotes.
> Those are the economics of the industry today, or not today but where we're projecting forward in a year or two.
OpenAI published their operating margins today. I can't find a non-paywalled source but Judd Legum reported it to be -122%.
There are huge economies to be had by batching requests and using lots of RAM for MoE (sparse models). You can't achieve that efficiency at batch size 1 on a single node.
If the arms race stopped tomorrow the current price pays for the inference.
also it's possible that the scale of inference needed (e.g. Jevons paradox) keeps growing to the point that training costs can fully be absorbed (since training cost is one off vs. inference that can scale).
(I suspect that might be the thinking, I don't know if it will be true, it's also possible that no model will create a moat big enough to attract enough of the inference traffic to make it true).
Depending on the chips/architecture used, the off-peak traffic from inference can also subsidize the training costs.
In other words, inference is fairly profitable for them and the rest of the money is spent growing revenue as quickly as possible. Building models is still an expensive line item but the costs for that are going down with time.
There is also maybe a “capture the market” mentality but I don’t think that’s necessarily it - the tools and processes are largely fungible and that’s a huge problem. They need to figure out how to make it sticky for “capture the market”, but there’s also a very real “grow as big as possible as quickly as possible to take on Google”; Google has an existential threat here.
They could have said the same about transistors. People keep inventing new ways to keep the costs down. Just look at the latest Qwen, DeepSeek, BitNet. Interesting tidbit: they’re all open, and as Google said in 2022: they have no moat.
How big/deep of a loss?
I feel like I read this every day for years that Uber did this same "idiotic, losing" strategy (how it was pitched/discussed) and then one day we woke up and... without much fuss, boom, they were profitable seemingly overnight.
For me nothing says low class like the Porsche dealer saying we can call Uber for you to take you home ridiculous… and it was a low class experience dirty car small never again ha ha ha…
Devil's advocate:
* inflation caused everything to go up to some degree since then
* if it was "that bad" as you say, they wouldn't be extremely profitable and have so many users
both things can be true? "they cut the driver pay in half and doubled the price" did not lead to the collapse of the business/people to stop using it.
Why? It's no less crazy than when Uber and Lyft were doing the same thing. Or when the entire tech industry was doing it in the dot com boom.
Investment-driven market growth at a loss is like the least surprising thing in all of this. The tech is new and fascinating. The bubble is just another trip through the funhouse.
The most intelligent model at a given time is much larger than the previous, which is why token costs for GPT5.5 are higher than 5.4. But you should expect that 2 years from now, serving a GPT5.5 sized model will be cheaper than GPT5.5 today. You should expect it to be even cheaper to get an equally intelligent model 2 years from now, because distillation techniques are effective at reducing the necessary parameter count for the same benchmark scores.
Seems to be on its way! I know of at least one person whose company is looking at a 20x increase, and afaict (from related looking around, nothing concrete tho) business accounts are missing some costs in the calculator so it'll likely be higher.
[0]: these API are not sold at a loss either, by the way. But it's a nice meme so let's just pretend they are.
Excusing everything else that u/bastawhiz said[0]; the obvious fact here is that Claude, OpenAI, Gemini et al. are quite literally burning through 100's of billions of dollars and selling it back to you for pennies on the dollar in the hopes that they get to be the only one left.
If I spend $10 growing Oranges and sell them to you for $1; then of course it's more expensive for you to do the growing.
I feel like I'm taking crazy pills. These models will become more expensive over time, it's functionally impossible for them not to, they just want to capture the market before they have to stop selling at a huge loss.
[0]: https://news.ycombinator.com/item?id=48168433