Without leveraging system RAM and/or SSDs, I don't think you can, or how exactly are you running this, if this is something you are doing today? With CPU/expert offloading you could probably do it with a 5090 + 1TB of RAM or something like that, but absolutely not on a single 5090 entirely within VRAM.
There are a lot of optimisations that are not in the public sphere, source working on start up in this space
Sure, but if we're participating in public discussions, isn't it more fun if we talk about things people can actually read and understand, rather than secret stuff other's can say work, but no can actually validate or know how it works?
It sounds like "hybrid approaches are much better than the public is aware, because everything else is private and secret", but also: ok, so what? No one can run that anyways, (yet?), so why it matters?
> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090.
Is painting a very different perspective, even considering the latter parts it's hard to read that as "Of course offloading everything else that doesn't fit on the GPU itself". But anyways, it's been clarified now so no harm :)
Because Kimi K2.6 in Q4 is about 584GB GGUF size on disk and will use slightly more than that in RAM, Q8 is 595GB.
> With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. Kimi K2.6 is released as INT4 already.
So 5090 with K2.6 is just gonna sit idle 99% of the time, waiting for next slice of weights to load.
5.6 Sol calculates that single 5090 in raw compute & memory bandwidth can run K2.6 at 35 t/s (256k context depth) -- if it somehow had enough memory to hold whole model in VRAM. Man, I hope HBF succeeds and Nvidia brings it to consumer cards in 5 years..
It's worse than that: a typical MoE model routes a separate set of experts at every layer, not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching.
You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability; though the obvious tradeoff is that having to store KV caches for the wider batches may leave you with less room to cache experts across layers and tokens.
(Plus if you're batching so widely that you end up loading essentially entire model layers, MTP then becomes applicable even for a MoE model. But this typically only applies if you're doing inference on a very large scale, or if your memory bandwidth is so scarce that you have to recover compute intensity by any means feasible.)
Caching really has nothing to do with this. With RAM offload you can mostly benefit from:
1) Batching for prefill is a huge win, even with MoE, since the batch sizes can be so large.
2) Keeping non-expert weights in VRAM, so the percentage of weights used per token in VRAM is higher. This benefit reduces with larger models, though.
> You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability;
With MoE it's low probability.
> MTP then becomes applicable even for a MoE model
With MTP it becomes _extremely_ low probability.
For even the sparsest MoE open models, having more than a handful of inferences in the batch is enough to make it more likely than not that you'll get some MoE weight reuse within any given layer. This assumes totally random sampling, ignoring any cross-request correlation that would push that probability even higher in many practical scenarios.
> With MTP it becomes _extremely_ low probability.
This is actually right, MTP is only ever worthwhile in very special cases involving either dense models or extremely wide batching of MoE ones that somehow still leaves unused room for parallelization (which AIUI would involve an assumption of very abundant compute with very limited memory bandwidth).
If you tell me the model and the number of parallel streams, I will do the math.
But my comment on running it was more towards serving this profitably at scale. You get much better throughput / unit of compute if you load everything in VRAM and serve many requests at the same time. That's how all inference providers are doing it.
The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!]
[!] With this and other MoE models it looks like an interesting area for research would be to detect or predict which models would be needed ahead of time. That way you could schedule the load into VRAM step before the weights are needed. That way you shouldn't lose much/any performance from offloading the weights to RAM.