News for hackers. Stuff that matters.
Re:Apple has new Mac app coming to make running local AI mod... by brainless on Wednesday November 11, 2026 @01:48AM (#50450397)
I really do not understand the value of running large models locally. I am absolutely fond of local models but for specific tasks. The agents need to do a lot of work in this case: orchestration, big or small model split of planning and work, split into small tasks, merging, etc.

LLMs with <30B parameter models seem like the sweet spot. I would rather focus on multiple tiny LLMs that are tuned to specific tasks (SQL queries, finance/accounting, backend/frontend dev, etc.).

For larger models, a $10-40 ish Neuralwatt/Opencode Go or any similar subscription goes so far for the average user that it is ridiculous. We are still in the "youth" era of agents. There are many patterns that work well with smaller LLMs. Or with a big-small LLM combination.

My context:

I own the tiny 16GB M4 Mac Mini and I use LLMs (coding agents) to build software and a few of my products use LLMs directly. Full-time user of LLMs, do not read or write code manually. I run the tiny Qwen series (<10B params) sometimes but a couple $20 subscriptions (Claude, Neuralwatt) take care of my full-time agent usage.

Re:Apple has new Mac app coming to make running local AI mod... by tokioyoyo on Wednesday November 11, 2026 @01:55AM (#50450442)
Model/harness that runs locally, which you can connect from your phone to do non-dev queries (looking stuff up, recommendations, planning and etc.) can save ~$300/year on various subscriptions.
Re:Apple has new Mac app coming to make running local AI mod... by brainless on Wednesday November 11, 2026 @02:05AM (#50450501)
Agreed but I feel we will see even cheaper annual or prepaid options. Say, a $20 upfront and get the cheapest inference, comparable to Sonnet with subscription. For a personal assistant that can run on laptop and phone, these will be mainstream. $20 may last a couple months for vast majority of users.
Re:Apple has new Mac app coming to make running local AI mod... by rapsey on Wednesday November 11, 2026 @01:57AM (#50450453)
The addressable market is enthusiasts. The cheapest and least loyal consumer that exists. But I guess it is just a small side project for apple which is fine.
Re:Apple has new Mac app coming to make running local AI mod... by conductr on Wednesday November 11, 2026 @02:28AM (#50450658)
Idk. Once they say our AI app is facilitating AI needs of big number of DAUs, the stock price will likely go up even if there’s no revenue path. It’s a pretty tulip.
Re:Apple has new Mac app coming to make running local AI mod... by jmaker on Wednesday November 11, 2026 @03:03AM (#50450853)
When I hear Apple and AI, I think of Siri. And then of my attempts to use it in my smart home where it would turn off all smart sockets instead of turning on the lights in one room. Or its response latency. Or how I was excited about Apple Intelligence and then removed it completely and am hesitant to upgrade to OS 27.

I understand that we’re talking about local inference with an Apple inference engine for Apple Silicon uniform memory. And I know it’s a stretch of an analogy. But somehow I’ve grown skeptical of Apple’s AI capabilities. Not to say Microsoft’s or Google’s weren’t disappointing.

Re:Apple has new Mac app coming to make running local AI mod... by conductr on Wednesday November 11, 2026 @10:57AM (#50454899)
For me Siri hasn’t improved a bit since release and it’s just the fact that she can’t understand my voice/words. Constantly mishears or asks me to repeat things. I have a pretty “regular” or “generic” US English accent and other voice agents have no problem so it’s definitely Siri. So I have never really been able to ascertain if it’s any good without an acceptable interface. All to say, the spirit of your analogy seems appropriate to me.
Re:Apple has new Mac app coming to make running local AI mod... by ThePinion on Wednesday November 11, 2026 @02:02AM (#50450486)
I have the same Mac M4 16gb. Do you have the Qwen model assist the larger models for menial tasks/tool calls during coding it can handle? As in have them work in tandem to save on tokens/calls on the large models or are your local agents just meant for their own specific side tasks?
Re:Apple has new Mac app coming to make running local AI mod... by brainless on Wednesday November 11, 2026 @02:10AM (#50450536)
A mix of both. I have been experimenting building agents that focus on small/tiny LLMs. For example, I built a web-app prototyping harness which used Qwen 4/9B models but separate agents (tool calls, prompts and actual code to parse) for CSS management, TSX management, Router, state, etc. It used SolidJS only.

There was a task manager which was given context of the tech stack so it could split the incoming requests. Then branch out the tasks to appropriate agents. Combine the generated code - this was a file/module/import manager.

The other way is let tiny models (again I would use agents for different use cases) do the busy work - tool calling, finding evidence, etc. I usually setup agents to re-write tool call history and create fresh API requests without using multi-turn chat with tool call history.

Re:Apple has new Mac app coming to make running local AI mod... by ThePinion on Wednesday November 11, 2026 @02:56PM (#50457812)
I really appreciate the response, that gives me good ideas on where to start going about all this. It seems so silly to have API models doing so much file exploration and tool calling.

I saw this a while ago which shows small models and focuses on the ones that make the least (or no) wrong tool calls and it got me intrigued on the possibilities in this realm, especially for lower end devices. https://github.com/MikeVeerman/tool-calling-benchmark

Re:Apple has new Mac app coming to make running local AI mod... by phyzix5761 on Wednesday November 11, 2026 @02:27AM (#50450649)
Privacy is a big reason to want to run model locally
Re:Apple has new Mac app coming to make running local AI mod... by brainless on Wednesday November 11, 2026 @02:31AM (#50450684)
True that. But ZDR is also not that expensive. Although for many businesses, buying a Mac with tons of memory is not expensive. Running a dedicated agent on top of private data is certainly valuable.
Re:Apple has new Mac app coming to make running local AI mod... by bauruine on Wednesday November 11, 2026 @03:14AM (#50450898)
ZDR is just a "bug" or "rogue AI agent" away from being all data retention. ZDR is not for privacy but for corporations that need to check a compliance box so they can blame someone else.
Re:Apple has new Mac app coming to make running local AI mod... by nolist_policy on Wednesday November 11, 2026 @03:44AM (#50451056)
Openrouter now does CSAM scanning and retention on so called "Zero Data Retention" requests.

https://openrouter.ai/blog/announcements/csam-screening/

Re:Apple has new Mac app coming to make running local AI mod... by Hamuko on Wednesday November 11, 2026 @02:43AM (#50450756)
[delayed]
Re:Apple has new Mac app coming to make running local AI mod... by WokeUp420 on Wednesday November 11, 2026 @02:39AM (#50450733)
It's called privacy
Re:Apple has new Mac app coming to make running local AI mod... by olowe on Wednesday November 11, 2026 @03:25AM (#50450942)
I can imagine a future where we look back at these times of relying on big opaque hosted services to run inference and find it quaint. Perhaps the same way nowadays we find it quaint that you couldn't always just compile software on a fanless portable computer.

How far away that is, or whether it will ever happen, I don't know. I agree though that the tooling is still so young it's hard to say what shape this type of computing will look like and therefore what kinds of resources will be in-demand.

Same context as yourself. I run 8 - 12Bish models locally on a 24GB M4 Macbook Air and also have a couple subscriptions (OpenCode Go, Mistral Pro) for access to DeepSeek etc. to power through yucky big bloated codebases at work.

Re:Apple has new Mac app coming to make running local AI mod... by yieldcrv on Wednesday November 11, 2026 @04:13AM (#50451222)
subagents use mostly the same allocation of memory, so you can do orchestration and planning easily with many subagents just like you do with Claude Code with anthropic's cloud models

Pi has a subagents package to enable in its harness