I understand that we’re talking about local inference with an Apple inference engine for Apple Silicon uniform memory. And I know it’s a stretch of an analogy. But somehow I’ve grown skeptical of Apple’s AI capabilities. Not to say Microsoft’s or Google’s weren’t disappointing.
There was a task manager which was given context of the tech stack so it could split the incoming requests. Then branch out the tasks to appropriate agents. Combine the generated code - this was a file/module/import manager.
The other way is let tiny models (again I would use agents for different use cases) do the busy work - tool calling, finding evidence, etc. I usually setup agents to re-write tool call history and create fresh API requests without using multi-turn chat with tool call history.
I saw this a while ago which shows small models and focuses on the ones that make the least (or no) wrong tool calls and it got me intrigued on the possibilities in this realm, especially for lower end devices. https://github.com/MikeVeerman/tool-calling-benchmark
How far away that is, or whether it will ever happen, I don't know. I agree though that the tooling is still so young it's hard to say what shape this type of computing will look like and therefore what kinds of resources will be in-demand.
Same context as yourself. I run 8 - 12Bish models locally on a 24GB M4 Macbook Air and also have a couple subscriptions (OpenCode Go, Mistral Pro) for access to DeepSeek etc. to power through yucky big bloated codebases at work.
Pi has a subagents package to enable in its harness
LLMs with <30B parameter models seem like the sweet spot. I would rather focus on multiple tiny LLMs that are tuned to specific tasks (SQL queries, finance/accounting, backend/frontend dev, etc.).
For larger models, a $10-40 ish Neuralwatt/Opencode Go or any similar subscription goes so far for the average user that it is ridiculous. We are still in the "youth" era of agents. There are many patterns that work well with smaller LLMs. Or with a big-small LLM combination.
My context:
I own the tiny 16GB M4 Mac Mini and I use LLMs (coding agents) to build software and a few of my products use LLMs directly. Full-time user of LLMs, do not read or write code manually. I run the tiny Qwen series (<10B params) sometimes but a couple $20 subscriptions (Claude, Neuralwatt) take care of my full-time agent usage.