But the problem is a $20 ChatGPT/Claude subscription utterly trounces it in usability, user experience, and performance. By the time we get Opus 5.5 at home, we'll be on like Opus 7 or 8, and I'd bet Sonnet 7 or 8 will dust whatever can be run on the base MacBook of that time. It's fine if people think local 5.5 will be good enough, but it just seems like as AI keeps getting better, you'd want to use the better tooling - nobody has decided a 2010 MacBook was good enough when the M1 came out.
I ran OpenClaw in a VM on my mac a few months back, only using local models. It was painfully slow (no MLX) but what it created was on-par with Claude at that time too.
The models are certainly powerful enough, but the frontier models are more than just an LLM model in the back.
- As AI continues to improve, the gains will become increasingly incremental. This means that your local model will be almost identical in performance as the latest cloud version, for all but edge case problems.
- If your local model is good enough for your current needs, the only tokens you are chewing through ae utility costs and asset depreciation. Relatively cheap. It is easy to overrun even 200 /month cloud models for serious AI use.
How much is "good enough" depends what we want it for.
Right now, the vision which was presented as a joke in (AFAICT more than one project all called) vibeOS seems like it could be viable with better local inference… but that likely needs a few more 9s of reliability even relative to current SOTA models, and we might even never get there if smaller local models always produce code which big hosted models can find vulnerabilities in.
Same if you want home inference to replace hiring a professional lawyer, architect, tutor, etc.: even though the AI has been getting passing grades in the law and medical exams since the early days, whoever gives out public liability insurance probably want to see more 9s of reliability before giving the OK.
Not with the tok/s I get.
When comparing API prices and privacy concerns, local might win.
20tk/s is not unlimited when you have a lifespan that isn't infinite.
Not good for PII/PPI
in 5yrs time running an M12 with a Bionic LLM Engine (or whatever they call it) may be good enough for a very very good AI experience. Enterprise / professionals will likely still pay for frontier models.
For starters, they need to let me get rid of their AI stuff and free up the space. Apple aggressively pushing their AI down the throat of their users means that they suck at it but trying to make it happen using their muscles in other areas.
Just like people figured out how to make their Android TV not suck, it suggests that some time it the future people can make their OS not suck and this will make macOS irrelevant if Apple doesn't fix it first. Unfortunately for Apple, they operate in a system where they must keep growing their numbers.
Such a great company, its a shame that they couldn't figure out this AI stuff. Fingers crossed that this tool actually fixes that by just giving ap and let people do whatever they want to do as efficiently as they can on the brilliant hardware Apple creates.
Especially in the modern era, the firmware/software side of macOS definitely doesn't seem like a priority.
I am super interested to see what the Apple silicon team can deliver when their full attention is turned to local LLM acceleration.