that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

Not to say the cost-cutting wouldn't be valuable; today's race is predominantly about the model's reasoning capacity or "how hard of a math problem can the model solve".

It'll be a nice day when research-oriented human capital gets redirected to things that benefit us layfolks's pockets more directly

So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

The downside is it is incredibly easy to be tricked by a single false bit of info.

So what should you cull and how much? There's already techniques in CNNs to trim unused or less active neuron paths to reduce a model's size, but how do you (and how much) do it in a general LLM? A product that will be used without supervision from a child to an 80yo elder?

MoE (mixture of experts)?

Now I have all of that information stored in a pointers-to-resources fashion, where layers of small "directories" of triggers-to-information point the agent gradually towards deeper, more esoteric knowledge the more specific its needs become when tackling a goal.

Write them in the margin of a book....

"It is impossible for a cube to be a sum of two cubes, a fourth power to be a sum of two fourth powers, or in general for any number that is a power greater than the second to be the sum of two like powers. I have discovered a truly remarkable proof, but this margin is too small to contain it."

Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

I think this can be achieved already. Take a base model and train only on source code. In fact, the very early Granite models from IBM were like that though it didn't support reasoning which limited its performance.

You can do it too. I don't know how much it will cost to train on just source code repos. $10K in total? Not sure.

We might think that knowledge from logc and discrete math would spill over to coding. Unfortunately, it doesn't seem to work like that. Even 1T parameter LLM fail on tasks if there are no variants of it in the training data.

to what extent is "retain the logical ability" meaningful without attaching it to some knowledge

Some knowledge is obviously required, I'm just less sure that a specific task like coding benefits all that much from having Shakespeare in the training set...

The A in 26B-A4B is the active weights.

The problem is that this is a per-token load/unload at best, not for the whole prompt.

The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.

It's still early days and we "just" don't really know how to do it well.

This looks as if you are just advertising.

Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.

What he's envisioning is a dense 1B model that looks at the Python specification and your prompt and goes:

> Ah, I get it now! This is like Darmok and Jalad at Tanagra!

Or at least:

> Android UI development in Python? It's UNIX, I know this!

To an extent we do work like this sometimes but in general we rely on internalized knowledge so I don't know to what extent it is a viable strategy.

That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.

There is just a stupendous amount of everything packed into a 35B size or larger model. For instance Qwen 3.6 35B A3B (Q8) can do a fairly decent job translating English to Arabic, but it can also generate python code with a reasonable layout and commenting.

I try to remember that as a mental model, an epub copy of a truly gargantuan sized 1000+ page novel such as the unabridged/2nd revision of Steven King's The Stand is about 800KB, and we're talking about a GGUF file that's 37GB in size or something like that.

The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.

And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol).

It's not about the people there being smart or not, it's about their and the company's priorities, resources and what they choose to focus on.

The second would be hardware savings on the order of tens of billions of dollars if they were stupid not do try all possible optimizations.

Spot the difference.

OP suggests that the token rate of of their solution is ~5 per second. That's at least an order of magnitude slower than commercially available models.

The new version of Apple Foundation Model (AFM) Core Advanced is an exception, it actually routes experts per prompt (with sporadic rerouting perhaps?) which is more in line with what you're proposing. But this will theoretically make the model less smart than a similar one where experts are picked per layer.

The results for English and Python are much better because the model is also trained on Mandarin and Greek and Lisp even if you never make a request or receive a response in Mandarin, Greek or Lisp.

Instead of taking syntactic shortcuts, the richer abstractions learned from multiple languages, and code, and math, and images and audio, ultimately make English comprehension and reasoning far stronger.

If you want citations, ask your favorite LLM how linguistic diversity prevents "surface memorization", overfitting on surface-level English patterns instead of representing the deeper concepts in latent vector space.

I think a rough analog is that it would be difficult to organize the rows of a hash table of everybody in a country by their geographic location.

At this point, people are just thrilled LLMs can even function as they do…

my 2c on King Charles vs 42.