That's hardly relevant. I can use any local model through providers such as OpenRouter and get blazing fast prompt response times, and you're charged a small of what an OpenAI/Anthropic prompt costs.
What makes local models extremely competitive is the fact you do not need a huge data center to run them. The hardware they require is far from impressive. Check Mac studio benchmarks and how some models output >200token/second.
Also, open-weights models are more expensive per task than Haiku and Luna
Yes they are, but you should understand what is the defining feature of a local model. Being able to run them locally is a consequence of that defining feature, not the defining feature. And it turns out that it makes them cheaper to run.
Moreover, you can run both local models and non-local open weights models through providers such as OpenRouter. They evem have Qwen2.5 7B.
> Also, open-weights models are more expensive per task than Haiku and Luna
You should really check your notes. Luna only remained competitive because OpenAI was forced to halve its cost just to avoid being dominated by open weight models. I seriously recommend you check out OpenRouter's benchmarks.
I checked. OpenRouter provides their own. They use their actual cost, which is what they charge users, and their benchmarks results. Please, spend a few minutes to browse it. You'll notice that the likes of Luna is dominated by quite a few entries.
We should check it in few months after IPO
With couple affordable GPUs you can get it to 200t/s. Macbook is not the best price efficient example. Tensor parallelism is the key, the same technology that they use in the cloud.
Ant moat is model. If model quality is not an issue for you, there are plenty of provides which will give you access to open models.
Are they? I'm on a macbook M5 pro 48GB (so, much better hardware than most users), and the speed alone is a problem, qwen 3.8 27b is ~10x slower than opus 5.5.
Capability-wise... For software engineering, qwen is nowhere near as good as opus 5.5. Might be fine if your usage of LLMs is close oversight, but it's not how I use them (I do review AI work closely and iterate, but I expect agents to autonomously do good work nowadays). I've not tested it much for "life assistant"-type things, but the speed would make it unbearable.