News for hackers. Stuff that matters.

A comment on Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac. Parent comment

Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R... by oezi on Wednesday July 29, 2026 @10:15PM (#49105422)
One key thing with MoE I am still not understanding is why we don't keep the same expert in memory for a larger number of tokens than 1. Why do we route to some other expert every token? Wouldn't it be more memory efficient to generate at least 2,3,4,8,12 or 32 tokens and then swap the experts?
[ Parent | on HN ]
Re:Show HN: Open-source engine running Gemma 4 26B in 2 GB R... by zozbot234 on Wednesday July 29, 2026 @10:28PM (#49105486)
Expert choice actually happens per layer, not just per token. It's not a limitation when doing inference at scale since all experts are then loaded in fast VRAM anyway. It's mostly just a way to enforce some kind of model sparsity and save on compute.

The new version of Apple Foundation Model (AFM) Core Advanced is an exception, it actually routes experts per prompt (with sporadic rerouting perhaps?) which is more in line with what you're proposing. But this will theoretically make the model less smart than a similar one where experts are picked per layer.

[ Parent | on HN ]