Re:Qwen3-Next
by furyofantares on Saturday September 13, 2025 @02:57AM
(#45229938)
> There is not way it's sending experts to the GPU per token.
Right, it seems like either experts are stable across sequential tokens fairly often, or there's more than 4 experts in memory and it's stable within the in-memory experts for sequential tokens fairly often, like the poster said.
I don't know how lmstudio works. I only know the fundamentals. There is not way it's sending experts to the GPU per token. Also, the CPU doesn't have much work to do. It's mostly waiting on memory.