If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."
ie what everyone asking for this fails to immediately realize.
Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.
This is roughly what multi-agent systems are built for.
This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.
There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.
Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.
I think I disagree. For some things, maybe that works - but think of a multi-agent system where one agent understands the code, and passes it off to the reasoning agent to figure out what the bug is. This system is going to suck. Because encoding enough info to figure out what the bug is would just be dumping every single line of the code.
So say agent 1 (reasoning) asks agent 2 (swift) to explain what is happening in File.swift. Anything agent 2 passes to agent 1 short of the entire code is a lossy transfer - and then the bug gets missed.
https://linux.die.net/man/1/ls
Hearing has volume, direction, pitch, it's spacial processing etc
I can change which one I hear on a whim, so much that I can even get Brain Needle and Green Storm out of this video.
THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).
When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.
> so LLM performance in swift benefits from pythonic patterns
What I hear you saying is that the best way to make a swift-trained-only LLM smarter is to train it on some python too. And then with an infinite parameter budget, every other programming language or really any other data you train it on makes the model smarter - I accept that premise.
But in a fixed parameter budget, what is better? training on 50% Swift + 50% Python, or 50% Swift + 50% Rust. Because if I am doing Swift programming, I want whichever of the two is better for Swift. If I am doing Rust programming, maybe I want the model trained on 50% Python + 50% Rust. Sure, it would be smarter if you tossed in the swift code too - but we have a budget to stick to.
Now is it possible to make those pluggable? i.e. can you take a model trained on 50% python, and layer on 50% rust OR swift depending on what language you're using? Probably not right now, but maybe one day?
And so far even the biggest model doesn't seem to have a working Make No Mistakes module, so maybe that's not needed
Sure, 3.8 maybe it's better now, but an accurate comparison would be with a new Gemma4-31B iteration (that doesn't exist).
Assuming you can tweak the training data, regenerate qwen3.6, and get a better coder, then presumably you could have variants - e.g. qwen3.6-swift-27b and qwen3.6-python-27b. Or maybe all coding is too intertwined and you can only get splits like qwen3.6-research-27b and qwen3.6-coding-27b. Which isn't quite my pluggable-models dream, but it's a step closer.
But maybe the difference isn't the training data, it's the architecture, in which case pluggable models is probably not possible.
Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.
Definitely not MCP, as that pulls info into the context. Unless contexts become REALLY big so that I can add 10B in swift knowledge, that's not gonna help me.
Possible RAG? I don't know enough about how that works, but I think that's not quite it either. I don't want to import facts like "the swift standard library contains a reverse array function", i more want to import knowledge - e.g. the parameters used to generate the text to reverse an array in swift.
Tool calling wouldn't do it either. You'd have to encode every single possible bit of useful info into the tool call, and the tool response would have to encode every piece as well (variable names, function scopes, types defined in other files, etc). E.g. how does it find a bug, if you have to pass understanding back and forth between the brain that understands debugging and the brain that understands THIS code?
I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly.
I think what you’re describing is the closest approximation we reasonably have right now though.
There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle.
> I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode.
And all for what? To piece together info that you can just query from a data source?
There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same.
Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.
Harnessing the Universal Geometry of Embeddings https://arxiv.org/abs/2505.12540
Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing.
Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models.
We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family.
And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests
sorry but no, you deserve a pop in the mouth for dropping this one.
- Understanding of protocols like HTTP.
- HTML, JS, CSS, SVG, and everything "web".
- Understanding of databases, SQL, etc.
- Abstract code architecture patterns.
- Understanding the users' requests in English.
- Responding in English.
- Command line tool usage (agents/harnesses)
- Industry-specific knowledge that can be applied.
- Frameworks, SDKs, applicable libraries.
- Relevant legal requirements.
- Etc...
It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please".
Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
Everyone assumes that carefully crafting a specific AI architecture with bits and pieces bolted together based on their human intuition is necessarily superior to simply using a bigger monolithic AI model. It turns out that the opposite is true, and has been demonstrated over and over again.
The bitter lesson is this: You can simply ask a frontier model to do the things you suggested, in a few terse lines of English. Dump a few lines in AGENTS.md and you are good to go.
Your approach is to "fiddle with inadequate tools" for weeks or months until you can finally attain a pale imitation of what the frontier models can do effortlessly.
It's the classic "But I can customise EMACS endlessly, why would I use an actual IDE?" argument all over.
I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
It's like LEGO or IKEA furniture: study after study has shown that people enjoy things more if they "put it together themselves", even if fundamentally the thing is worse and/or still essentially nothing more than plastic made in a factory.
You don't _have_ ownership of someone else's ai, and that comes with real risks.
Security risks, privacy risks, business risk.
They might rug pull you, they might charge you more, or like atrophic, silently corrupt the answers, or code...
The labs are happy to jump on any emergent capability the scaling and training impart: generate prose, teach you things, cyber security, design, code, etc.
Do you really think that the frontier labs won't turn a popular capability, or trend they notice, into a first party tool if the ROI seems there? If it's your own private ai in your datacenter, you can keep it all secret, and not lose your business.
On the bitter lesson you're right of course:), wish I had a super computer to just scale that instead.
A) You can always self-host something like Kimi, DeepSeek, or GLM.
B) Just because you use a specific proprietary AI for programming doesn't actually bind you to that provider in any meaningful way. The authored code remains even if you stop paying them!
Of course, if you use AI as an active component in some sort of service, then the EULA, rug-pulls, etc... suddenly start to matter. That's a different story.
I think I agree with the bitter lesson, but I wish I weren't :)
> A) You can always self-host something like Kimi, DeepSeek, or GLM.
I mean, one could rent-a-box for, like, 10$$ per hour? Agentic loop development gets really expensive at scale with larger models, like, if you want to A/B test two tool schemas to see which works better, and you run 100 benchmarks...
But why are you so convinced the bitter lesson is true, and it's not just a temporary lead? Proper agent loops and RLVR are like, 3 years old at this point? At some point, right, the compute can't scale it out further? And at _that_ point the lead position might go back to: highest compute + smartest designed smarts.
Against my somewhat better judgement I'm currently "assembling small AI pieces" :(, for lack of access to unrestricted models for offensive security work and, ehh, funds. It's _okay_ so far, I'm running private benchmarks and look at the trajectories. To be perfectly honest, qwen is _really_ doing well, finishing quite complex chains without a lot of smarts in the prompt. Just "Go pwn {server}, use these {tools}".
But, there are some smarts embedded in those tools. Helpful errors, retries, benchmarked/handy representation. Strict validation of what the model tries to do, etc.
> The authored code remains even if you stop paying them!
This is true, but my point is that they will outcompete you if you happen to stumble on something that actually makes good money using LLMs and becomes popular. Obviously, if you don't then they won't.
> Of course, if you use AI as an active component in some sort of service, then the EULA, rug-pulls, etc... suddenly start to matter. That's a different story.
That's the plan hehe
That paragraph sets me off. I’ll take Vim and Emacs over VSCode and Eclipse any day.
> Visual Studio Code (commonly referred to as VS Code)[11] is an integrated development environment
Unless you were just going for a sick burn on vscode, in which case carry on :)
There’s no need to bring religion into this.
And of course the neural network series.
You just defined a liberal arts education.
Or yesterday, when I asked it for the best low resource approach to select the matching string from a collection of strings for a shell interface it suggested an exhaustive string compare over the entire collection. Because it was thinking of how David Lynch would tackle it.
I think you're missing the point of the article though, which suggests not to mix reasoning capacities with actual knowledge. Sure working with Swift and Python is basically the same, it's programming, with the same concepts etc. Much closer than the ability to drive a car. But the point is, the methods, libs etc are all different and things are changing each time a new version of the language is released. Like you don't need to relearn how to drive a car if you go to the UK but you have to known the wheel is on the right. Knowledge shouldn't be stored in the weights.
The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role).
This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”.
You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do.
I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.
Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher.
The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.
Chomsky would like to have a word. Your statement is true only at a surface leve l( they have nouns, verbs and some limits), but it breaks down the moment you start to inspect it closely:
- sign language ( I am not being petty; you put all languages ) - it is almost nothing like the underlying structure of other spoken languages primarily because it does not carry its restrictions - English vs Polish example - word can carry grammar or not; word order can carry meaning or not
Those are just two examples, but both clearly show that little about human languages is actually the same. It is kinda like the history thing. It rhymes.
I am addressing this part as other posters noted issues with other parts.
I think this claim is acceptable in context, but taken alone, this needs to be qualified. Some aspects of language are universal, like abstract information structure and other pragmatics, but no model is going to speak rural Khmer dialects well anytime soon because not enough of it has been digitized, fundamentally speaking, hence qualifying what it means to "be good at all languages".
Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!
And then ideally, make it pluggable so I can pick what I want from off the shelf components, but if that's not possible - then just train up as many variants as you can so we can all pick the best variant for our current need.
Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used.
Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage.
(This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied)
So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context.
Reasoning about language itself to its full meaning is quite hard.
Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist?
So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive?
I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.
I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.
I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.
I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.
The vast majority of people listen to the music of Ed Sheeran and think that it is good.
I think you're right that current architectures don't compose like that - but I feel like that's a result of the focus on MOAR DATA, and a "race for AGI" - if we set those ideas aside, a more composable architecture seems very possible.
No. It means that the one model has 2.4 trillion parameters while the other has only 27 billion. I don't know the details about their architecture or training, but presumably they used the same or similar training sets for both and a conceptually similar architecture, scaled down. I'd guess they also have some techniques to re-use some of the work done for the big model for the smaller versions (if anyone knows more about this I'd be interested). The architectures cannot be identical by definition because then the parameter count would be the same. Subsetting the data to such narrow fields as you describe could risk losing some edge, there are a lot of emergent capabilities in those models and I don't think that emergence is fully understood yet. There are subject-specific models, but for far broader subject areas than you suggested, like coding or math or prose.
I'm sure composability is possible in principle, I'm just sceptical that it'll be a good long-term solution, for my originally stated reason. It's basically just The Bitter Lesson again, we may gain some short-lived edge by putting more domain knowledge into the algorithm, but ultimately (these days often: surprisingly quickly) it'll be outgunned by something that just leverages raw computation better.
The issue for me is that the raw computation is coming at the cost of the planet. Throwing an aircraft carrier at a problem that needs a bicycle is dumb, but because the damage to the environment required to scale up computation isn't included in the price of that computation - it's easier to just toss the aircraft carrier at every little problem.
So when I say I want to pick and choose, and use smaller models, it's because I like technology and I don't want to hate LLMs, but I also like the planet and don't want LLMs to continue to accelerate environmental collapse.
If you're running Qwen3.8-27B on energy-efficient hardware like a Mac or a DGX Spark instead of an API (likely running on H100s), I'm sure you're having much more of an impact than you would by switching to, say, a 9B coding-only model on the same hardware. The thing is, I think you won't be able to go orders of magnitude smaller, because a lot of the usefulness of LLMs comes from emergent smartness, and you typically need a minimum amount of complexity to see such emergent phenomena (and I think we're pretty far from understanding this kind of emergence, much further than from the next model generation that annihilates the current one on benchmarks yet again).
I'd always thought we'd eventually hotload loras or MoE experts.
It would certainly be useful on the robotics/VLA side of things as well; more limited mobile hardware, download and load/unload new skills as needed.
Tbf I also don't really care what facts my models have baked in (for llms at least). I care most that the model understands general logic and then general knowledge of some level is secondary. Reason being is that everything is RAG'd in anyway.
Models spitting out well established facts is cute but I don't really ever want to rely on say "electronics knowledge" that exists in a tenuous and vague form in the model weights.
Humans write books (and datasheets) for a reason. Books are RAG.
Right now we have the Star Trek computer. A polymath of knowledge. But I’m sure you’re right. It’s the trend of all knowing to move from the bazaars to the temples, and back again.
So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.
Then when I want to research electronics components, I grab a 15B model of agentic research techniques, and add in 10B of electronics knowledge, etc.
I don't want general purpose models. They try to be everything to everyone. I want to click together a model that is laser-focused on what I am doing, and I want to run it locally