Re:Ornith-1.5: From Self-Scaffolding to Self-Improvement
by slim on Wednesday August 19, 2026 @08:58PM
(#49369188)
llama.cpp can run MoE with some layers in vram and some layers in ram
Re:Ornith-1.5: From Self-Scaffolding to Self-Improvement
by dannyw on Thursday August 20, 2026 @03:52AM
(#49371658)
Expert offloading significantly helps with the VRAM capacity.
Most MoE architectures have a few experts that are always running; this, the router, KV, and whatever else you have space for can stay in fast VRAM; and the remaining experts can be offloaded.
That's not true. For computers without unified memory architecture (which is the vast majority) VRAM capacity is the bottleneck for local models. In that case a dense model can deliver significantly more intelligence than an MoE model of the same size. And for a typical consumer/gamer Nvidia GPU, dense models are fast enough.