I ran OpenClaw in a VM on my mac a few months back, only using local models. It was painfully slow (no MLX) but what it created was on-par with Claude at that time too.
The models are certainly powerful enough, but the frontier models are more than just an LLM model in the back.
- As AI continues to improve, the gains will become increasingly incremental. This means that your local model will be almost identical in performance as the latest cloud version, for all but edge case problems.
- If your local model is good enough for your current needs, the only tokens you are chewing through ae utility costs and asset depreciation. Relatively cheap. It is easy to overrun even 200 /month cloud models for serious AI use.
How much is "good enough" depends what we want it for.
Right now, the vision which was presented as a joke in (AFAICT more than one project all called) vibeOS seems like it could be viable with better local inference… but that likely needs a few more 9s of reliability even relative to current SOTA models, and we might even never get there if smaller local models always produce code which big hosted models can find vulnerabilities in.
Same if you want home inference to replace hiring a professional lawyer, architect, tutor, etc.: even though the AI has been getting passing grades in the law and medical exams since the early days, whoever gives out public liability insurance probably want to see more 9s of reliability before giving the OK.
Not with the tok/s I get.
When comparing API prices and privacy concerns, local might win.
20tk/s is not unlimited when you have a lifespan that isn't infinite.
Not good for PII/PPI
in 5yrs time running an M12 with a Bionic LLM Engine (or whatever they call it) may be good enough for a very very good AI experience. Enterprise / professionals will likely still pay for frontier models.
But the problem is a $20 ChatGPT/Claude subscription utterly trounces it in usability, user experience, and performance. By the time we get Opus 5.5 at home, we'll be on like Opus 7 or 8, and I'd bet Sonnet 7 or 8 will dust whatever can be run on the base MacBook of that time. It's fine if people think local 5.5 will be good enough, but it just seems like as AI keeps getting better, you'd want to use the better tooling - nobody has decided a 2010 MacBook was good enough when the M1 came out.