I foresee the same thing as IOT devices. Yeah, you can run everything locally… but few people actually do.

Everything is proprietary clouds, unpredictable support horizons, bricked devices and monthly subscriptions

> Local models are already extremely good - good enough for most normal users

i would love for that to be the case but i think you're completely wrong. setting aside the question of what "good enough" even means in this case, i don't think a normal user has a machine capable of running a decent local model with any sort of acceptable throughput.

I think the models themselves are "good enough", but people are underestimating how much the integration with tools effects the quality. And OpenAI/Anthropic are lightyears ahead there.

I don’t think these integrations are much of a moat at all, unless I misunderstood your point.

Pi and OpenCode are just as capable as Claude Code when it comes to integrations?

Fair counterpoint. But people still use Windows. The market is huge for people who want a subscription and a ready-to-go solution.

Very true. I think for your run of the mill user they’ll reign supreme. But for companies on Bedrock/OpenRouter already, I wouldn’t be surprised if they switched to open weight and open source harnesses.

You can easily hook local models into CLaude Code e.g

Considering most corporations are the ones who buy the most tokens, this is relevant because:

- They can afford to dish out hardware to run models locally - They can afford to deploy even larger open weight models on their own infra, even if they’re not local

In addition, the writing is on the wall. These open weight models are getting good enough and small enough with each iteration. Hardware will (hopefully) come down in price eventually so a layman could deploy these locally.

As AI boosters love to say: [Local models are] “the worst they’ll ever be right now!”

Everyone was moving to the Cloud well before AI, why would they go back to on-prem now that hardware is even more expensive?

I’m not saying or thinking they’d start buying up bare metal. By hardware I meant issuing MacBooks with the latest chips or with bigger GPUs.

By infra I more just meant it generally. Whether it’s bare metal, cloud, Bedrock/OpenRouter etc.

I agree, I think the bigger threat to them is the neoclouds serving flash models like GLM 5.3 flash or DS 4.1 flash 1017, those are very good, very fast, and very cheap. With zero data retention and less guardrails they become an obvious choice for more work, and enterprises can post train on top of them.

That being said, their own small models (Haiku/Luna) have gotten very good as well..

I really don't see why by this time next year most people won't be paying pennies for models in this tier and getting the performance we'd expect from current Opus/Sol models, e.g. good enough for 95% of what you need

My guess is that at that point open weigths will become illegal to download and own..

*chinese open weights models will become ilegal.

Of course there won't be any US open weight model of a high quality.

Good thing we spent several decades perfecting the distribution of things that are illegal to download then

local models are economical pipe dream. Before open router, you have to buy GPU commitment for any large usage. essentially doing that 24 hours every day in a year.

Why on earth anyone would use local model, and absorb all the costs, if you can just pay $20-$100/mo and not deal with any hustle? Unless you are developer, or someone who use it extensively, but in this case, local model is not enough for you.

If your use case is trivial you pay extra price for hardware while using mediocre quality. If your use case is intensive , e.g software, science, medicine, legal, etc, you want cutting edge. You getting it for free, immediately.

Sure, but as "local" models improve, I've heard of more and more companies deciding to bring it in house. As hardware prices possibly get more attractive, that will make more and more sense for many companies. Enterprise don't mind paying flat rate for SaaS but variable billing model gives a ton of business people pause.

I'm an SRE and I just wrote some terraform to deploy G4DN instances for local evaluation for mid-size company.

[delayed]

Open models seem to catch up in spurts. There was short period where suddenly a bunch of Chinese models came out that were on the verge of catching up, like GLM 5.3, Qwen 3.8, Deepseek 4.1, Kimi K3, and so on, but then Opus 5.5 and now Fable 5.2 seem to be pulling away again.

I don't think they will ever overtake because they are probably using distillation, however once the open models pass the "good enough" threshold it won't matter.

See, users are quite price insensitive for models below the threshold, no matter how many tokens you throw at Opus 4.8 it will never be as good as Opus 5.5 because of compounding errors. Above the threshold, price becomes hugely important because results come from throwing more tokens at a problem rather than using a better model.

> Local models are already extremely good - good enough for most normal users

Are they? I'm on a macbook M5 pro 48GB (so, much better hardware than most users), and the speed alone is a problem, qwen 3.8 27b is ~10x slower than opus 5.5.

Capability-wise... For software engineering, qwen is nowhere near as good as opus 5.5. Might be fine if your usage of LLMs is close oversight, but it's not how I use them (I do review AI work closely and iterate, but I expect agents to autonomously do good work nowadays). I've not tested it much for "life assistant"-type things, but the speed would make it unbearable.

> Are they? I'm on a macbook M5 pro 48GB (so, much better hardware than most users), and the speed alone is a problem, qwen 3.8 27b is ~10x slower than opus 5.5.

That's hardly relevant. I can use any local model through providers such as OpenRouter and get blazing fast prompt response times, and you're charged a small of what an OpenAI/Anthropic prompt costs.

What makes local models extremely competitive is the fact you do not need a huge data center to run them. The hardware they require is far from impressive. Check Mac studio benchmarks and how some models output >200token/second.

Local and open-weights are different things

Also, open-weights models are more expensive per task than Haiku and Luna

> Local and open-weights are different things

Yes they are, but you should understand what is the defining feature of a local model. Being able to run them locally is a consequence of that defining feature, not the defining feature. And it turns out that it makes them cheaper to run.

Moreover, you can run both local models and non-local open weights models through providers such as OpenRouter. They evem have Qwen2.5 7B.

> Also, open-weights models are more expensive per task than Haiku and Luna

You should really check your notes. Luna only remained competitive because OpenAI was forced to halve its cost just to avoid being dominated by open weight models. I seriously recommend you check out OpenRouter's benchmarks.

I seriously recommend you check out the cost-per-task Pareto frontier, which not a single open model is on

https://artificialanalysis.ai/

> I seriously recommend you check out the cost-per-task Pareto frontier.

I checked. OpenRouter provides their own. They use their actual cost, which is what they charge users, and their benchmarks results. Please, spend a few minutes to browse it. You'll notice that the likes of Luna is dominated by quite a few entries.

Subscriptions are actual costs to users. And Openrouter doesn't consider that. If you're talking API prices, those aren't actual costs to inference providers.

does not consider subscription subsidization

[delayed]

Compared with subscriptions, I don't think open models are that cheap. You can get an incredible amounts of API dollars for $20.

until it is being subsidized.

We should check it in few months after IPO

[delayed]

Only need $10k worth of hardware!

> macbook M5 pro 48GB

With couple affordable GPUs you can get it to 200t/s. Macbook is not the best price efficient example. Tensor parallelism is the key, the same technology that they use in the cloud.

So instead of buying a $20 Anthropic subscription you recommend that the average person go out and buy a couple affordable GPUs and build a home server?

Once the memory cartel is reigned in by CXMT, Apple ships a Home with 512GB unified memory for <$10k. It'll have confidential compute so they'll let you sell tokens and they'll collect 30% for facilitating.

> So instead of buying a $20 Anthropic subscription you recommend that the average person go out and buy a couple affordable GPUs and build a home server?

Ant moat is model. If model quality is not an issue for you, there are plenty of provides which will give you access to open models.

I looked into this last week, and it seemed like if I wanted to get something in the Opus 5.5 range I'd be talking about $500k in hardware. Is this actually all much cheaper than I was seeing?

The weights of Kimi K3.1 Pro are 1.56 TB. I guess with four of the upcoming MaixBlade cards from Sipeed you'll be able to run it without quantization but that setup would still be >$50k.

What do you call a couple affordable GPUs?

what quant and engine? i have the same hardware and i get 40tps with a 4bit quant using omlx + splash. which is about the same as opus, or faster, maybe? with its caching the time to first token is also very impressive.

Have you tried the open source splash inference engine? I have the same setup as you and find it fast enough for most tasks.

https://inco.ai/blog/splash/

https://github.com/incoai/splash

Anthropic's spend commitments at half a trillion are unbelievable. I love their product but think they've leveraged up awfully hard... makes me wonder how quickly software engineering, marketing, and other valuable features will soon be siloed and locked behind increasingly lucrative paywalls.

In a roundabout way, I hypothesize about Anthropic's future pricing power by using the analogy of historical cable TV package pricing compared with the total price that households are now paying for various streaming services. Cable TV was often $100+ per month in the 2000s when Netflix came along and offered most of a viewer's favorite shows plus movies and eventually originals for $7/month, streamed at any rate a viewer wanted. Over the years since we've watched media companies snatch back their most valuable IP and start their own streaming services, and the result is households now pay a total cost that approaches what people used to pay monthly for cable TV packages.

There's little doubt we're in an era of very cheap AI, as most people are well aware given the amount of money Anthropic and OpenAI are losing. I also hope local inference can help suppress the future pricing power of Anthropic and OpenAI and continue to enable somewhat equal access to all of the benefits of AI in the future.

I thought this, and ~4 months ago ordered a M5 Mac Studio with 128 GB RAM. I've been experimenting quite a lot with local open weight models, including but not limited to DeepSeek V4 Flash, V4.1 Flash (the latest one), various Qwens, and a couple of others.

What I'm seeing is that although some open models are capable enough for some coding work, they're way behind Codex and Claude even on being able to reliably complete tasks. They need quite a lot more managing and steering, and sometimes you need to retry a prompt several times to get the result you're looking for.

That's not to say they're not useful - far from it. But what I'm also seeing over the couple of months I've been experimenting is there actually seems to be a capabilities ceiling that's not really going higher for the same equivalent (local) compute, even as new open weights models are released. I think Qwen 4's delay is a signal here too.

They're still roughly in that 4-6 month lag time, though I think the lag is slightly closing over time.

Great for general purpose, still not competitive with e.g. gpt-6.1-sol at the price and speed frontier

which are those "normal users" that have capabilities to install, run and maintain local models to be somewhat at the same level as enterprise models? Not to mention RAM and hardware shortages caused by AI model companies. Level of delusion is unreal

>> Local models are already extremely good - good enough for most normal users

What is "good enough" and what is a "normal user"?

>Local models are already extremely good - good enough for most normal users.

Not really accurate. Normal users aren't going to go through the trouble of setting up a local model and sure aren't going to be willing to spend the money required to actually host a good model at usable speeds. Your average tech nerd buying kitted out Mac Studios with 128gb of ram are nowhere near a "normal" user.

Open models yes (eg Deepseek Flash) but local models are not, at least with attainable hardware. My brand new Mac Mini M5 Pro 64GB can run Qwen3.6 at a reasonable speed but it's nothing compared to hosted models and likely never will be.

That said I'm optimistic that in time we will at least get to something close to DS locally and if that's true I could see a lot of dev work being done locally, at least for more clear cut tasks.

> we will see AI appliances that can do everything most users want, and privately.

we _could_ see that but would we really? you think? Since when do companies these days make the best experience they could for the consumer? Instead of spying / upselling / advertising etc.. while providing the bare minimum needed to hook them into the initial purchase

Alibaba has Qwen 4 models ready to be published once IPO start.

Local models are never going to be a thing outside of hobbyists. Look at how many people don't switch to Linux (a completely free OS) because they don't want to deal with the hassle of installing it. Telling people that they should pay more for their computer to go through the hassle of installing an LLM is an even harder sell.

Great point for personal use. But what about organisations, products and service providers. Most business infra runs on Linux.

Locally hosted models for an entire organization means having to hire people to install and troubleshoot the LLMs. You wouldn't want the employees to goof around unmonitored, so the LLMs would need to be centrally located and accessed through an internal portal, which means buying a couple of GPUs. Besides that, there's something to be said about having a company to blame if things go wrong, as opposed to blaming yourself if there's an issue.

>Local models are never going to be a thing outside of hobbyists.

Never is a long time. Think about 10 years from now when ai models fit on your phone. Some already do...

Just my anecdote but I use Instinct heavily (personal assistant AI who can do computer use tasks like buying flights or scheduling). It now runs a surprising amount of my life, but it messes simple things up like getting me completely accurate notes about the person I have a meeting with. My understanding is they are using GPT Luna behind the scenes, so it’s hard for me to think something that is small enough to run locally and is not at a frontier lab would be able to catch up (for now).

Separately, I am not sure these things are maximally useful with guaranteed privacy. They need access to other data sources (banking, logins, business data lake, observability stack) to be useful for things outside of pure coding or research.

I don't think people want to run their own models on their own hardware. They want a service that's available, up, and constantly updated to be the best. What you are talking about seems like a niche model to me. Of course we'll have local models that can do everything but we want a service that can just remember stuff. I can access it from multiple devices and it just works. People will pay good money for that because the value is insanely great.