The moat right now is a) the hardware, b) the electricity, c) the intelligence, and d) scalability.
On hardware, it's very expensive to purchase anything which can provide a fraction of the performance of a subscription. Traditional accounting depreciation would imply that purchasing local hardware is a terrible financial decision.
On electricity, this is a surprising cost center depending on location. A system with just one 5090 can easily pull 1kW, and to achieve usable performance for a workplace is going to require dozens of machines. This can represent an extra $10-20k in electricity in cheap places. In California or Europe this could be $30-60k per year.
As for intelligence, the frontier models from OpenAI and Anthropic are still superior, and they have at least a 3-6 month head start. Distilled models are closing the gap on some metrics, but they still can't compete. That's why they cost so much less.
The last major moat is the ability for subscriptions to scale with need. This means easily adding and removing licenses. This is far easier than purchasing extremely expensive hardware (and managing it), and selling it if/when internal demand changes. It's the same reason companies use contractors. The ramp up/down costs are very high.
The only real moat that local LLMs have right now is privacy.
All it takes is one incident, and all your company's internal data will start showing up in public users' chats. You can rely on a contract to prevent this, or you can guarantee it by using a locally hosted model you fully control.
When combined with the cost savings and good enough performance mentioned in the article, this can become a huge selling point.
Corporate doesn't care at all about privacy. It's why the run outlook and windows and let Microsoft scoop up all of their company secrets. It's why they hand every scrap of data they have on their customers to salesforce and surrender their data to Atlassian and use Confluence and JIRA over countless alternatives.
All companies care about is that when data breaches happen publicly they can point the finger at someone else.
And with things like Fable mandating data retention at Anthropic it makes things business records which US Federal Administrations have a history of obtaining without warrants and under mandatory gag orders like national security letters. They generally won't issue those directly to law firms (because they can and will fight), but IT companies generally comply quietly and have government contracts to think about.
If however, our internal IT group didn't read the manual correctly and set up our cloud service access parameters incorrectly so that anyone who was trying to probe our defenses could access the formula, Jim from IT is looking for a new job.
See the difference?
You'd never hear about if it the cloud service was taking your corporate data and using it for countless other things though. Insider trading, selling leads to your competitors, pushing ads to your clients/customers with offers of their own products, using trends in what your company and others are doing to guide development of their own products and services. Done carefully, you'd never be able to trace any of it back to them and it'd take a whistleblower for you to find out it happened in the first place.
I don't imagine any cloud service that has massive amounts of data which could easily make them a ton of money is just ignoring it.
I doubt they want to risk their reputation, trust and agreements to make money in the ways you describe.
OpenAI and Anthropic are fighting to win a race (build the biggest baddest model) that has no prize. The prize is mass adoption at scale at the best price, which is why companies are rapidly shifting to open model. They don’t need to pay 10x for a model that’s provides no practical additional benefit.
> As for intelligence, the frontier models from OpenAI and Anthropic are still superior, and they have at least a 3-6 month head start. Distilled models are closing the gap on some metrics, but they still can't compete. That's why they cost so much less.
I would argue that the reason they cost so little is because anyone can run open models and offer them as a service, so there's actual competition and the price is closer to cost. i.e. if the open models were just as intelligent as frontier models but cost the same to run as they do right now, the price wouldn't be higher (unless demand went up so high that marginal cost to provide more of the service went up, due to scarcity of hardware and or electricicy).
On the other hand, if what you're saying is the frontier labs have some pricing power due to their models being better, and that is the reason they are able to charge more than the companies providing open models as a service, then I would agree.
One of the basic questions/concerns here though is that it's not like the AI places are getting the GPUs for 10x less. It's true they have some economies of scale, but they also have some waste, and frankly in this particular case it's not clear they get that much gain over what a lot of businesses could achieve. The biggest traditional gain for central providers is that a lot of typical computing usage is burst-y, and in turn local kit might be underutilized. But with LLMs heavy users tend to use them all the time assuming their tokens allow it (and in the case of local hardware there's nothing stopping you, quite the contrary), they can use it directly interactively or leave them to go overnight on something too.
So it's reasonable to suspect that the reason subscriptions are only a fraction of the cost is that we're in a bubble seeing these companies losing money in an attempt to gain some sort of durable advantage. Just as every previous time, there is the chance that the music stops at some point, and they need to crank up pricing or pull other schemes to actually make money. Of course, it can be a good deal in the mean time, you basically get to suck down investor money for nothing, but it's also not unreasonable to at least be consider fallbacks. Even beyond questions of control and risk etc. I know at least a few places that are now genuinely considering questions like "what happens if a datacenter we depend on gets droned" that would have never had an iota of thought devoted to them even 5 years ago.
>On electricity, this is a surprising cost center depending on location. A system with just one 5090 can easily pull 1kW, and to achieve usable performance for a workplace is going to require dozens of machines. This can represent an extra $10-20k in electricity in cheap places.
I don't think that's "surprising" at all, everyone knows about power use. And this seems like it gets heavily into what you're defining as "usable" and is also more useful to define in terms of cost-per-employee vs total. Obviously a bigger business will have a higher line number total even if the cost per employee is identical, but simultaneously can be expected to be making more revenue to pay for it.
If we're defining an average of a dedicated 5090 pulling 1 kW for every single employee (presumably some people wouldn't use it all the time, but others would then pull the compute for other work), running 24/7 (to cover people running stuff when they're away), then that'd be 8760 kWh per year. At my not particularly cheap New England location that'd be about $1900 per employee per year at the generalized residential rate (~$0.22/kWh), or $156 per month. That doesn't seem radical if it really does boost productivity. However, there is a lot of room to go lower. I'd expect a business to run backup anyway, and these days there are a lot of incentives to do that at least partially with batteries. That also opens up rate shifting as another way to pay back the cost. If we change to time of day pricing, that's 8 hours of peak pricing with the rest off-peak. 8 kWh of battery can now be had for a few thousand. And the off-peak rate is only ~$0.14/kWh, cutting the cost per year by about $700 to $1200 per employee per year. Solar power is also usually far more valuable to use yourself then sell back to the grid, and also continues to plummet in price.
None of this is to say that it makes sense for every place at all, but it's close enough to the the line that the math is at least worth exploring, or could at least lower the cost enough to be worth it given other things. It really comes down to how much extra value the company (or individual) expects to come out of it per month.
>In California or Europe this could be $30-60k per year.
Dunno about Europe, but at the kinda prices I see for California I'm really surprised more places aren't trying to move a lot of usage to battery+renewable.
>The only real moat that local LLMs have right now is privacy.
I don't think resiliency and control are things that can be taken for granted anymore, particularly on the global scale. War and terrorism is getting worse again. International relations are getting nastier, and governments have the power to just order places cut off. If LLMs aren't particularly valuable to a business, then why an expensive subscription? But if they are particularly valuable, then insurance is something leadership should be contemplating.
I'll grant they are superior at least right now. But also, they are too expensive.
We ($work) are finding that it is best to build engineering discipline around AI usage (who would've thought!) and use the cheaper models like Cursor Composer.
Using Opus we can blow through an entire month budget in an afternoon, so while more powerful, it is no longer practical except for rare very complex tasks.
OpenAI and Anthropic rent their compute from AWS & friends. When we say large enterprises are moving to open weight models it means they are cutting out the middleman and renting the compute directly from AWS instead of giving OpenAI and Anthropic a margin.
> they have at least a 3-6 month head start
This is a moat of nothing. Our company still hasn’t gotten access to Fable so switching to open weight models would mean getting access to similar quality models. In some orgs they are still on 2025 models.
Depreciation is designed to _encourage_ purchasing of useful local tools, by incrementally matching fractions of the cost of the tool to the revenue it generates over its useful life. The fact that a graphics card might have a book value of $0 after five years of depreciation is a feature, not a bug.
Since the invention of corporation tax it has also had the benefit of offsetting tax over the same period, instead of just one big offset in the first year.
I'm not challenging the concept of depreciation. It's a necessary tax function. I'm explaining the business case for local LLMs is poor.
The important part is if they break the contract or raise their prices you can always move to a different provider. You get lower cost and lower risk at the same time which is extremely rare in business. That's just not possible for closed models where your only options are the official branded API or Azure/Bedrock.
It's the same with the subscriptions. Local models can't compete because they're simply giving too much value for money. They're effectively subsidised by Big AI. Again something that won't last.
If the companies survive the next few years, which they probably will because they represent too much of US economic growth to allow them to fail, this gap will keep on expanding.
Starting from zero without distillation is a lot harder, a lot more expensive and a lot more work. OSS models is what a laggard does to get adoption. China's gov't might keep on sponsoring it as a counter GPU embargo thing, but when gov't get involved, usually the other side gets involved too.
As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.
One way or another they're getting the same results as proprietary labs, with a fraction of the hardware for a fraction of the cost. OpenAI can't keep raising funding rounds of $100billions to subsidize their compute costs and get results by brute forcing parameter count. And if they're having trouble keeping up with Chinese labs' efficiency, maybe they should stop worrying about distilling and instead hire some of the smart people responsible.
To actually have something competitive and improved within the next 3 months and not be perpetually behind, you need your own independent model creation process. So to extend the metaphor, a complete movie studio with cameras, actors, staff, sets, budgets, etc. It's the right strategic move to do when you are GPU constrained, which the Chinese labs are, but it won't let you get past it.
A bunch of pedantic people will come out of the wood work citing a bunch of things saying that is not the case because of some detailed mechanics of how model training works and they will get fixated on some of the words I used, but zoom out to the level of what an AI lab is able to produce and this becomes evident.
Per the article, companies are dropping OpenAI+Anthropic (partly) because of costs. If distilling is so simple and easy, why doesn't OpenAI take this "quick shortcut" and serve a self-distilled model externally, so they can charge reasonable prices and stop bleeding customers? Surely they can at least match the Chinese labs' efficiency, right? Wouldn't more customers and less opex look good for the IPO?
The open weight labs figured out some secret sauce that (so far) big name labs are unable to replicate, so instead of competing, they're going on the defensive with claims of distillation attacks.
Chinese AI labs do well because they have the best AI people coming out of a huge talent pool.
[There are many examples of distillation attacks.](https://cyberpress.org/anthropic-claude-ai-distillation-atta...) Of course, you could argue this is just a form of "learning" from other intelligent systems, and this is arguably what LLMs have been doing from the beginning. So I don't begrudge the Chinese labs doing it - they all do it. But let us not pretend that it's not happening.
Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right?
If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability. Try asking K2 Horizon to write iambic pentameter for instance, or even give it the car wash prompt. I tried both these on the Q8 quant for the 7B model and the results were depressing.
> As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.
To help you further understand, the Chinese labs will never admit they were distilling until they are better than US labs with their own non-distilling process or some sort of espionage-like public reveal shows it. So you need to look at secondary indicators. Much like how the chinese government lies about their economic stats so 3rd parties use secondary indicators to figure it out.
Nathan Lambert has made this distinction explicitly: he thinks Chinese labs likely innovate heavily on distillation, while saying he wouldn't call it a crucial factor in their post-training capabilities, in part because the RL work still has to be done by the lab itself -- just like what I said above.
https://www.interconnects.ai/p/how-much-does-distillation-re...
They’re burning cash like there’s no tomorrow. They desperately need to show that they have a real business and not just a giant burning pile of cash doing academically interesting things. If they could simply sell models that are 95% as good at 1/10th the price they’d do that. They’re losing the enterprise sector because they’ve not done that.
It certainly is possible that Anthropic and OpenAI are doomed to bankruptcy, but without a known drop in revenue it seems exceptionally confident to make such a certain claim.
1. Competitors to OpenAI and Anthropic are good because of distillation.
2. OpenAI and Anthropic will come up with some methods for preventing distillation in the future.
Both of these are dubious imo. For RLVR tasks like coding in particular, you definitely don’t need continuous distillation to improve, otherwise OpenAI and Anthropic themselves would not be able to improve because there is no better model to distill from.
But Claude also makes it really hard to do that, so what am I even really paying for? Time to extract all my data, put it into a sqlite with FTS5 and make sure I never rely on the overly-opinionated, low-thinking PMs from these giant orgs again.
Of course, that "easy" step has lots of partial solutions like CTK (Conversation Toolkit) or MyChatArchive and I haven't found the perfect one yet, ideally it'd be something that dumped everything into Obsidian or an Obsidian-alike, but surely somebody is working on that? I'd pay $5/month for somebody to solve that problem for me, as long as I still owned the data...
I’d try:
1 exporting my data (I imagine it’s common outside of GDPR?)
2 asking Claude to convert it to an easily digestible format :)
What I'm doing instead: syncing coding agent sessions to a central backup location, and for cloud LLM chat providers I'm occasionally exporting data.
Still need something to automate that syncing, and provide search and viewing.
Don't rely on chat history. Have it write and maintain summary files that you can import into different sessions, at least for anything important.
Over time I can imagine us becoming mostly open models on our deployments when hardware is more accessible and the need for expensive frontier models is constrained to very few use-cases that might demand their capabilities.
Just too many products to exhaustively list, it's a big tech company, we're trialing a lot under the sun to find workflows, tools, integrations, including a lot of bespoke internal research, there's a large ML department since almost the inception of the company.
Maybe there is some threshold where the price/quality math for your standard business tips in favor of smaller models and self-hosting the entire stack. I'd certainly love that.
But the "open" models are right there too, and I'll be using them more in the next weeks as I get better chat interfaces.
It's not about money, it's about control. Companies have lots of money and want control over their key technology.
The same dynamics that define the public cloud ecosystem are at play here. What AWS sells you is access to appropriate hardware and turnkey infra for your needs. Looking at the cloud industry over the last 20 years, I find it hard to believe that it is impossible to build a moat or a huge business around this.
The problem is serving is a skill readily mastered by the hyperscalers. That's their MO.
All they need is weights to serve. And the open models provide that.
OpenAI is relatively well placed in that they have inference chips they've designed and they own compute.
https://cloud.google.com/blog/topics/hybrid-cloud/gemini-is-...
I think there will be separate and huge markets for models, hardware and compute. That will maximize competition and innovation.
Why? Because even Blind Freddy can see the huge usefulness and power of these (and future non-LLM) models and no-one in their right mind is interested in becoming OpenAI's or Anthropic's bitch. Those companies have tickets on themselves.
Given the recent behavior of tech companies and the US administration, no one trusts either anymore.
I couldn’t care less which company drilled for the oil… it’s all the same to me. Models are increasingly no different.
OpenAI and Anthropic are a gas station saying “buy our gas for 10x the price!” When the world is looking at them saying it’s just gas, we’ll take the cheaper brand. We’ve tested your gas and it’s really no better than the stuff that’s 1/10th the price.
Thats why their present business plan is screwed.
I agree 90% of the world can work with 87 gas, but there's always niche/luxury market where 93 can make small difference.
(edit: typo)
The crazy setup here is that even with that fraction of the pie these companies might be worth say $100 billion optimistically, which would be amazing in normal times. Problem is it’s a train wreck for their investors and the associated debt bubble if they can’t sustain a valuation of 1-2 trillion and the present setup does not put them on a course to that trajectory.
When your competition has a tiny cost base compared to yours and lacks the bonkers future capital commits you made then that’s a terrible position to be in… hence their conundrum.
Take Open AI for instance: it has "zero debt" ... and $665 billion to $1.4 trillion in "long-term commitments".
For LLMs the costs of training and inference are a very significant part of the overall costs.
1. People buy Apple because of the broader ecosystem of products and the “it just works” aspect of that ecosystem. Other companies make phones with features that are objectively better but folks don’t switch because the Apple ecosystem is sticky.
Despite trying, neither OpenAI nor Anthropic has managed to move up the stack beyond “hey guys new model release today!” announcements that everyone yawns at.
2. Switching costs are real. It’s a PITA to switch not just the phone but everything else. Switching model providers is a line of code and takes almost no effort.
Apple has a true moat which is why they can command a premium. OpenAI and Anthropic have no moat which is why they’re in trouble.
Lots of people use old iPhones and don't care about some "up to" benchmark bumped every year, but are stuck with iMessage contacts, their stuff in iCloud, Apple Watch or apps that are not allowed by Apple to even mention they have Android versions.
It’s literally the least stickiest thing in the history of tech. Which is a big problem for these companies.
"To build or purchase the physical hardware required to store tens of petabytes of data and train a State-of-the-Art (SOTA) frontier AI model, you are looking at a capital expenditure (CapEx) ranging from $320 million to well over $1 billion."
They were set up as a public benefit company and their returns were capped at 100x. They (well, Sam) went out of their way to put themselves in this death march to the IPO. If they had just done what Mark Zuckerberg did with Muse, they're not in this position.
Do you know how bad you have to be at the tech business to make Mark Zuckerberg look like a prudent-yet-visionary leader?
You're giving him too much credit if you think he figured anything on his own - this is the guy who thought 'metaverse' was a good idea.
The job of a CEO is to have a vision that is compelling for the market and an ability to execute upon it, thus the "executive" in "Chief Executive Officer". Zuckerberg doesn't have this. What he does have is an absolute majority of voting control over Meta's shares, so his actual ability to do his job doesn't matter.
It's almost as if creating incentivizing systems that don't hold people to account is a bad idea.
Even that guy could read the situation.
> Unless they both dramatically slash prices then they’re in big trouble
False, they have already done so many times.
> Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any hope at a successful IPO.
False, margins are higher and I can have a formal bet that prices will go lower.
> However the cold reality for both is that there is zero moat to a model anymore
False, LLMs are not fungible and there exists a natural moat. I like the behaviour of Fable, not the behaviour of Opus - the fact that many people speak about this is evidence.
Not a tech company though, so maybe that differs and we’re just behind what’s en vogue. However we did get in on everything pretty early, building our ChatGPT RAG clone right about when azure got gpt-3.5-turbo on api
We’re heading into corporate budget season for 2027 when all this is coming under a huge microscope in boardroom after boardroom across the country at a terrible time for companies trying to IPO.
I'm wondering how much of that is the harness vs the model. Overall, the 'feel' of a model seems to be largely due to the harness than the model itself.
Is this a typo? Have you actually made a formal bet on a prediction market or something to put your money where your mouth is, or are you just saying that you could? There's a lot of things I could plausibly make bets on, but that doesn't mean that they're likely to happen.
Classism is the new racism.
Not everybody is coding or doing work that lends itself to burning tokens for warmth. Reuters for example seems to be more interested in using it for research, editing and formatting citations and the like. There's only so much of that work that needs doing, it doesn't always need to be real-time, and they probably don't see it scaling exponentially. They also need to be very aware and in control of their model's biases, or they risk it compromising their work output.
It's widely expected that all of the major providers will need to - and surely want to - drastically raise prices to justify the ludicrous amount of capital they're burning. Multiple companies have already talked about how their AI costs have exploded, and from what I understand that scale of enterprise is paying API rates. I would be disappointed if big business wasn't having a think about what that liability could look like. It's one thing to be reliant on a relatively "stable" vendor like Microsoft for Windows and Office, another to get AWS sticker shock, and then this is promising to be an order of magnitude worse.
Then just plain trust. What if ChatGPT starts recommending your competitors products, or the USA bars export of Anthropic's latest model (again, but for real this time), or they stop serving a model your business now depends on, and so on... That's a lot of risk to leave outside of your control.
Keep in mind that a vanishingly small number of workers are SWE's churning millions of tokens daily.
We've all seen the office spaces where there's 200 empty computers on a floor. Combined, it's something like 500 cores at ~3 Ghz each and around 3 TB of RAM. The networking is already there and software like exo already exists.
Think about how crazy it is for non-US companies to use American AI providers - their marketing boasts that you can treat their models as co-workers, assign tasks, invite them to slack annd video calls, etc. Taken at face value, would you hire someone remote who lived in a country that commonly does random shit like deciding whether or not remote workers aren’t allowed to go to work?
Having trained on your own chips, that is the impressive part.
Unless they both dramatically slash prices then they’re in big trouble. Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any hope at a successful IPO.
However the cold reality for both is that there is zero moat to a model anymore. It’s a pure commodity. Those selling compute and access to open models are gearing up to wipe the floor with Open AI and Anthropic.