Re:Small Models Have Arrived
by BobbyTables2 on Thursday August 27, 2026 @11:04PM
(#49473883)
It’s kinda funny that your last paragraph is basically describing why sparse files, sparse matrices, etc. are used in other contexts.
It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.
Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…
We certainly would know where they were if they were just in smaller models in the first place!
And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.