Not to say the cost-cutting wouldn't be valuable; today's race is predominantly about the model's reasoning capacity or "how hard of a math problem can the model solve".
It'll be a nice day when research-oriented human capital gets redirected to things that benefit us layfolks's pockets more directly
So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.
The downside is it is incredibly easy to be tricked by a single false bit of info.
So what should you cull and how much? There's already techniques in CNNs to trim unused or less active neuron paths to reduce a model's size, but how do you (and how much) do it in a general LLM? A product that will be used without supervision from a child to an 80yo elder?
MoE (mixture of experts)?
Now I have all of that information stored in a pointers-to-resources fashion, where layers of small "directories" of triggers-to-information point the agent gradually towards deeper, more esoteric knowledge the more specific its needs become when tackling a goal.
Write them in the margin of a book....
"It is impossible for a cube to be a sum of two cubes, a fourth power to be a sum of two fourth powers, or in general for any number that is a power greater than the second to be the sum of two like powers. I have discovered a truly remarkable proof, but this margin is too small to contain it."
that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.