Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R...
by subarctic on Wednesday July 29, 2026 @08:11PM
(#49104691)
I know very little about this but it seems like the kind of thing that can and eventually will be solved computationally, not by people figuring out what a parameter or group of parameters represent
Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R...
by bastawhiz on Thursday July 30, 2026 @06:15PM
(#49116572)
It's not a compute problem. It's a knowledge problem. Even if you can process each parameter individually and re-link the model layers, you need enough information to know what each parameter is for which is necessarily more memory than the weights themselves. You can use the weights to know whether each parameter is useful for a given prompt, but that operation is a strict superset of just generating the answer. By the time you know which parameters are useful, you've already done all the work of generating your output tokens and the effort is pointless.
Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R...
by jz391 on Thursday July 30, 2026 @12:33AM
(#49106144)
Would an extension to `madvise` to say: "page this region in/out as a whole" help here? Engine could define memory ranges representing each expert and leave paging to the OS (tho' "paging" at this point becomes more similar to swapping in granularity...).
That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.