Reminds me of linux gaming 10-15 years ago. According the to the community everything worked great without issue and any issues were hand waved away as tiny bumps that you won't even notice.

Then you would go try it and be back using windows to game a week later.

But that is not exactly a linux issue. Games built for linux works very well on linux like games built for windows running decently well on windows. Linux can also run some games built for windows - not for linux. And that are the ones that doesn't provide first rate experience. Those windows games executables technically should not even load as it is not built for linux.

Everything considered, linux is good and has decent capabilities including gaming and whatever you can throw at it but only do oranges to oranges comparison. Games and apps built for each os works fine on the respective os, well atleast it is true for mac and linux. Windows is very unreliable.

32 would get you a good Qwen 3.8 27b which is like Opus 4.6.

Memory manufacturers are running 80-90% margins. Any of a number of things (anti-trust, CXMT, financial shock) have the potential to drop memory cost by 2/3s. At the same time memory efficiency of models is improving. At some point Apple's going to start shipping 64G of unified memory in mainstream machines that are advertised as local AI capable (like HP, Dell and others do today). Like the iPhone refined the concept of a smartphone for the mainstream, they could be the first to hit mainstream sovereign AI.

The people saying that local models can easily replace the real Frontier Models from Anthropic or OpenAI are just not using it for the same things you and I do.

They seem to be happy with their small usages.

For everything I do, i see huge differences in Quality and Speed.

Even when I try an open model on a 4090, the PC gets warm/hot very fast, I struggle with context length.

> The people saying that local models can easily replace the real Frontier Models from Anthropic or OpenAI are just not using it for the same things you and I do.

> They seem to be happy with their small usages.

Oh yes, we're terribly inferior people. Just living on scraps from the table.

anyone who is working on an remotely difficult codebase will notice a huge difference, either you are an incompetent dev or dont code at all

Ahh yes. Those are the only two options.

I really strongly agree with this statement. A bunch of people aren't even engineers, and their abilities aren't that good either, yet they keep saying that open-source models can replace OpenAI or Anthropic's models.

If that were really the case, basically nobody would spend so much money using them.

Putting aside that you're "really strongly" agreeing with a personally directed insult:

You should factor in that undeniably most people don't have the means to run even a 27B dense model locally, and therefore do not actually know what they are and are not capable of.

And you might want to look at openrouter's rankings.

If you are comfortable assuming that "doesn't use Claude or Codex" is equivalent to "not a good engineer" or "bad engineer", then I think you are probably lost to more nuanced discussions.

I try local models regularly on 2x 4090 and struggle with context length constantly. 48GB on GPU and 128GB from RAM.

the original comment mentioned only 16GB of VRAM.

You know talk about Open Router and 27B dense model?

We are not comparing an even more expensive Hardwaresetup to run local models with proper context window and parameters.

Can i run some open model on a super expensive Mac Book or co with some kind of okayisch results? Yeah sure but you know what the t/s will be and yeah Frontier is still Frontier and Kimi, GLM, DeepSeek are not Frontier. They lag half a year.

You are not getting Opus 5 or Sonnet at home for now.

Why are you so offended by this?

I'm not offended. I'm also not the person resorting to insults.

With my DGX Spark + 5080 + 3060 setup, I've tried all kinds of approaches, including quantization, acceleration, fine-tuning, post-training, and everything else.

It still can't compare to Opus 5.5 max in terms of speed, capability, and stability.

And when I calculate all the money I've invested, directly subscribing to the service would actually be cheaper, more efficient, and more stable.

Open source is great, but don't glorify it.

The architecture of the code I develop has never been simple. If you're someone who doesn't even work on things that need to be deployed to something like k8s, then basically, open-source models are indeed enough for you.

You can sell your DGX Spark for basically no loss. Maybe even make money.

It wasn't meant as a dismissive comment.

I do plenty of experiments of things i always wanted to do and they just don't work with local models.

Finetuning of some ML models for a specific usecase, some 4k demo idea i had, small browser games to show others game mechanic ideas i had for ages, analysis tools etc.

Even just playing around with some Ghidra eats context like nothing.

can you give example, prompt require context window > local model, or higher Quality/Speed, that local model can not do, open source example?

Any program I'm vibing.

I write 5-20 paragraphs of things I want and it goes away and just builds it.

Something like this (just an example):

"Build an application with a webserver in rust. It has 4 stages, first one is an input module... second one is refinement ... third one is an image detection with current fitting ml model ... last one is finetuning."

Then i would describe every basic page with a few more paragraphs.

This would work for half an hour or an hour, consume 100-400k of context tokens and then I would test it, write down 10-50 small sentences (every sentence either fixing a bug, or making it better) which tehn takes again like 20 minutes and i'm nearly done with a complex enough tool providing real value and actually working.

A big kimi, which is hard to run, might be getting close to this but we do not talk about running it on 1-2 5090.

The other thing is, thats more like a practical experience, even if you use them through Open Router (for testing and co), the initial results of something like Opus 5.5 are ALWAYS better. More details, better quality, better understanding of usability of the tool.

I might be misunderstanding but I do the same thing with sonnet, and it runs faster and does well enough

The discussion was about local models vs. frontier. I would argue that Sonnet is frontier?

And yes Sonnet 5.5 is crazy good

Here's a prompt I used recently:

> In a work tree build a feature where each column cog action menu has a "Column statistics" item - clicking that item opens a modal dialog which displays statistics about that column - for text columns that's most common values, stats about value length - for numeric columns it's more interesting than that. Let's do a bit of percentile analysis in there too. Under the hood it works by firing off SQL queries using SQLite window functions and suchlike. For the moment let's have that option only available if SQL queries are allowed against the database, and the SQL queries are built in JavaScript.

You can see a screenshot of what Opus 5.5 built here: https://github.com/simonw/datasette/pull/3038

This:

Having currently a very tech-intensive chat running for more than one week, this is where the frontier models really shine: Its no more 2024 or so, where they lost context on large discussions, you can do really great work with the frontier models - the local models will be a thing in a few years, if RAM & CPU developed further, this will happen: in 2035 like, you will be able to run "Claude-like experience" on your iPhone.

12GB GPU is useless. The good experience is with 128-256GB unified memory architectures like strix halo etc. You don't need to buy any hardware, you can experience the latest open weight MOE (flash) models from DeepSeek, qwen, or GLM on openrouter using a local agent like kilo.

That is a shift of comparisons though, right? This thread was about running locally, explicitly as an alternative to cloud services. I would guess few people have graphics cards with > 12GB ram.

A few more will have Macs with 48GB RAM though. (Indeed if you push it a little, 36GB unified is possibly enough for Muse Glimmer; I have not tested this theory)

There is, for sure, a limiting factor in the x86 architecture, but I think the RTX Spark will change that a bit.

AMD Ryzen AI/Strix Halo is for running locally on your hardware. Openrouter is for evaluating the models you'll be running locally, before buying the hardware.

> 128-256GB unified memory systems running MOE models is the "cheap"

I'm not familiar with these. How much does one cost, and from what store / what product name?

Would you talk to me a mo about Kilo compared to Pi?

I am digging in on Pi (in lovely Paseo and exploring Pendant when in VS Code).

What would I find better or worse in Kilo?

A 3060 + good chunk of system RAM can run qwen3.8 flash next.

12GB is not a lot, you are stuck on low quantisations of older models with that. What exactly did you try?

The newer/larger models are good though, not as good as Claude, but not every task requires a model of that level. Maybe yours do, but for simple coding tasks, summarising text, etc a small local model runs just fine, and is a lot cheaper than Claude

What graphics cards have more than 12GB, and how much does it cost to buy one? What about for others saying you ideally want 32GB? (That split will determine if a 5080 will do it)

32GB would probably get you qwen3.8-27b at a reasonable quantisation, which benchmarks similar to opus 4.6

You can try that model (and many others) using a cloud host first, and then only if you like it do you need to buy something to run it.

> What graphics cards have more than 12GB, and how much does it cost to buy one?

RTX 5070 Ti and RTX 5080 both have 16 GB.

RTX 5090 has 32 GB.

Good luck finding any of them for reasonable prices, though. I paid $3,000 for my 5090 early last year. I winced when I bought it then. Today, it's like $7,000, and I wouldn't even consider it at that price.

But I'm mostly a gamer that also does local AI as a hobby. If the RTX 5090 was $7,000, I'd still be on my RTX 3080 and still just not started doing local AI and have to turn down the graphics settings in Cyberpunk 2077.

there is also a version of the 5060ti woth 16GB. i got lucky a few months ago and got it for ~380€, but i think you can still get it for around 800, definitely under 1k

The RTX 5090 is the ideal, but it's not the only 32GB GPU that can offer workable performance; the AMD R9700 is catching up.

You can surely build an LLM server with an R9700 and 32GB of mainboard RAM for less than the 64GB M5 Max Apple Studio, which is probably the next most competitive model up. It'll be better than a Strix Halo but less flexible.

Of interest:

  - 4090: Going for $7500
  - 64GB M5 Max: Starts at $3,800
  - R9700: $2700

Compared to $20 or $100 per month.

Right. Nobody says it's necessarily economically the best option.

But it's also an asset that barely depreciates!

Especially right now — you could buy and extensively use an M5 Max for a year and sell it for almost exactly the same amount at the end. Maybe even more, at the current rate.

Furthermore, it can do other things for you: host your VMs, edit high def video, whatever. Even if it stops being useful to serve LLMs, it might be useful to run as a server for other tasks for a whole decade.

Depends what you want, doesn't it? Or are we all expected to conform?

For me personally, right now I have an M1 Max macbook pro, and that is my local AI machine, because my needs are exploratory and I want to run it here. I want to learn by comparing models. I want to learn how to run them and advise others how to (local inference is not a zero-sized market).

My next step up will be to take the apps I'm looking at building that run against these small models, and host those apps online with some sort of cloud inference — openrouter, digitalocean, cloudflare. Again — if I can avoid it, my apps won't develop a dependency on any AI provider.

If I ever find I want to vaguely describe some desktop application before bed and come down the next morning to an aggravating slop version of it, I'll consider Claude.

As for a local machine, I've looked at the M5 Max and the R9700; haven't pushed the button yet because I don't need it enough.

I've also looked at the M5 Pro, which in practice is about as fast at token generation as my M1 Max, and smokes it at prefill. That alone might be enough for a "leave it running" mini AI box, because agentic coding is so prefill-heavy. But I am a laptop person for the next six months or so I think.

As an aside there's that other thing which is somewhat interesting, though constrained:

https://www.olares.com/one

Uses the 5090 mobile chip, which is pretty good speedwise (about the same as the R9700) but 24GB. Small, tidy, on-prem. I think they have missed the boat on pricing, but if they can ship in volume, maybe not.

Great insights! And I'm kicking myself for not buying a 4090 3 years ago for $1600, shipped direct from nvidia... it seemed like too much to pay for a GPU at the time!

I know a couple of people who have eminently LLM-capable GPUs more or less doing nothing when they are at work...

The assumption that is probably incorrect is that the current prices will last. A 4090 should not cost that much, and most of your Claude usage is paid by an investor. Neither of these things will be true in a few years.

Having a 4090 at home running agents all day is not a $20pcm level of usage, so take the $100 plan. Looking at the S-1 filing, today they are not even close to breakeven. It is reasonable to assume that will become a $300-400 pcm plan.

The launch MSRP of a 4090 was $1600. There has been significant inflation since then, so say $2000. Eventually supply will meet demand and you'll be able to buy one for that price again, or lower due to the economies of scale they must have now.

So, in 2-3 years time once it has all settled, you pay off your 4090 by saving 5-6 months of Claude usage. Maybe the local model isn't good enough for complete replacement, but all it needs to do is drop you to the $20 plan to be worth it. I think this is a large part of why investors are getting jittery now.

12GB GPU is insufficient for most purposes yes. Even 16GB is a notable improvement, but yes 32GB cards such as the 5090 or R9700 are a world of difference.

Given we're discussing a cost comparison (i.e. cutting a Claude subscription in lieu of running locally), how many months of claude $100 or $20/month plans would it cost to buy a 5090? I am showing it as 88 and 440 respectively. I.e. a 7 and 36 year payoff. Not even close to being worth it.

The Claude plans are probably still heavily subsidized. Expect prices to increase, especially after the IPO. Self-hosting models for personal use is probably overkill, but I can see companies doing it pretty soon since they can probably run 24/7 workloads and fully utilize the hardware.

Great point. And, 3 years ago, a 4090 cost $1600, so maybe that will happen again some day. (Same/similar card, or a future card. I remember when GPUs used to get better every few years instead of staying roughly the same but costing several times the original amount!)

Currently, it's not worth it because:

"SemiAnalysis, a research and analysis company specializing in the semiconductor and AI industries, found that a $200/month subscription plan is far more generous than previously thought. According to its calculations, while it's widely believed that a $200/month plan maxes out at roughly $2,000/month in tokens (assuming API pricing), the actual numbers are closer to $14,000 for ChatGPT and $8,000 for Claude. "

https://cybernews.com/ai-news/chatgpt-claude-usage-costs-ope...

This will change in the future.

You also get to keep your data private (so you should compare it to api pricing), there are no forbidden topics (i just did a pentest on my webapp), and multiple people can use it. Basalt does 150 tok/s on a 5080

As amazing? I suppose not.

Good enough? Quite probably, unless you're in the "coding is solved, specify loops and walk away" frame of mind.

32GB (or say a 48GB-unified RAM unified machine) gets you Qwen 3.8 27B at a good adaptive Q4, maybe Q5. Muse Glimmer at Q4 and Q5, etc.

These models can code well enough in an agentic way, I find, and the chat-to-assistant scenario is just fine in Unsloth Desktop or LM Studio.

A slightly lighter harness helps; a lighter system prompt is entirely possible, especially for Muse Glimmer, which is post-trained so effectively for agentic helper tasks that it barely seems to need a system prompt at all except for a "plan" mode.

Tool use is all fine, all that.

> unless you're in the "coding is solved, specify loops and walk away" frame of mind

which is where the entire software industry is going

Well it's certainly the service-dependent (addicted!) model that the big two want.

But will you want to do that when your tokens are metered at non-subsidised prices, and yet there is absolutely no clarity (ever!) about token efficiency, how the model decides to spend your money, how it might ever decide to stop wasting your money and get to the point, etc.?

The specify-loops-and-walk-away model absolutely points to local/on-premises/private cloud AI.

Because otherwise you're letting Anthropic or OpenAI decide how to spend your money. And they are going to be biased towards spending more of it.

I don't think Qwen 3.8 is quite there yet with the loops stuff. Definitely seems to be better at deeper and deeper one-shot puzzles.

Who cares what the big 2 want? Have you taken a look at how much the hardware/algorithms/efficiency are improving? Those 2 are building sand castles in front of a tsunami.

uhhh mission critical software is nowhere near turning over full control to _any_ LLM...

Have you tried Codex? My journey so far

- 2023 using Cursor

- 2024 using Cursor with Agent mode

- 2025 using Claude Code

- 2026 using Codex with Astra 6

I've try every new model when it comes out from both OpenAI and Anthropic. I cannot justify leaving Codex and Astra 6. The token burn on Anthropic is always astronomical for me. Codex just works, and I see no reason to move away from it.

I switched from Codex to Claude Code w/ Opus 5.5, and it's been fine. I haven't hit limits in my $100/mo plan, doing full time dev work all day. But I agree. Codex sub + Astra is also great. Both are excellent options, and I don't see a reason to go with a cheaper / less capable model, frankly.

This matches my experience, Claude limits with Fable blows up the entire 20x usage in like an hour, whereas Astra can run a couple days.

> The results are ALWAYS terrible compared to CLaude Code.

How much effort did you spend to make the local setup work well?

Local models are not plug and play, you need to adjust settings to make them behave intelligently.

On my side one model looking super dumb right after install ended up being 'agentic' after using the right chat template, fixing how tool calls are setup, fine-tuning plenty of params values, and adjusting the whole setup to my own hardware. Claude.ai (free) was great at helping me sort that all out.

You can't replicate Opus 5.5 at home.

You can maybe replicate Opus 4.5 to 4.7 if you have some moderately expensive hardware. And that is definitely good enough for many people.

The smallest model that can even pretend to be a decent coding model is Qwen3.8 27B, which you realistically need 32GB of VRAM to run without breaking or limiting it in some really noticeable way. This gets you something like a slightly less capable Opus 4.5 with a much better work ethic.

Strata and Qwen 3.8 flash next is an 180B model running at 70 tps on my 5070 ti and 64gb of DDR4 RAM at Q3 quants and so far I can't tell much difference between it and any other non local model for my workflows/coding needs. Not saying local is better all the time, but it's certainly a lot more capable with minimal hardware than you're giving it credit for. Qwen 4.0 is right around the corner too.

The problem is that "can't tell much difference" is not going to be an objective indicator; you will be very likely to overlook the flaws of the local model and overweigh the flaws of the non-local model. The only reliable way to compare the two is to have your own set of evals based on the types of tasks you regularly perform and then run those evals on Qwen and a frontier model.

I can't really do anything with this comment without a couple of specifics, such as 'what model are you using?'.

In general, more memory = larger models = more intelligence.

However even if you had a private data center you wouldn't be able to beat Opus 5.5, Astra or Gemini 4. The open weights models are a tier below these three.

So for a sense of scale, un-quantized 1 parameter == 1 byte of ram. 12Gb means you could have roughly 12b parameters, frontier models like Kimi k3 has 3 trillion parameters, so you'd need ~3tb of vram.

Imo, this is just the tech version of virtue signaling. Everyone knows local models are objectively just worse than frontier/cloud closed source models for most commodity hardware setups, yet people insist on using it because they want to be contrarian or "stick it to the man."

Your experience is average because it _is_ supposed to be average. The people using local models tradeoff taking the moral/technological high ground vs actually using something high quality to get a job done quickly and with high quality, and largely autonomously. They are ok with subpar output because they will handhold the model more, or the outputs are totally sufficient for their toy use case while being cheap.

Almost no one at any job where software is the main economic output is using a local model to code.

> If you actually require intelligence to get a hard job done, you should just use a frontier model.

Or, you know, your own intelligence.

What I do not understand as an engineer is how our community has gone from believing in open source for controlling destiny, security, etc., from believing in heterogeneity and portability, to just saying, look, if you're not using the product of one of two loss-making AI unicorns to do all of your work, you're just being a contrarian.

I guess if that is your definition of contrarian — that everyone else is doing it so why aren't we? — then I am.

But in my experience it's not so different to "why aren't you on social media?"

>Sure I only have a 12GB GPU

Found your issue.

I really strongly agree with this statement. A bunch of people aren't even engineers, and their abilities aren't that good either, yet they keep saying that open-source models can replace OpenAI or Anthropic's models.

If that were really the case, basically nobody would spend so much money using them.

I think the key to making it make sense is that people have wildly different demands for quality. You will see people say that these small local models have improved quickly and are "good enough for most work". While these models have improved and very fast, it still might be more of a comment on their expectations than the product.

12 GB isn't a lot. You need a lot of VRAM or RAM for context window.

Because local models are significantly worse? I'm not so sure about what your question is. What you got is normal. What else did you expect? If you could just build Claude Code at home, Anthropic would be valued at closer to $1.1M than to $1.1T.