Re:Ggml.ai joins Hugging Face to ensure the long-term progre...
by zargon on Friday February 20, 2026 @11:58PM
(#47097653)
A large model (100B+, the more the better) may be acceptable at 2-bit quantization, depending on the task. But not a small model. Especially not for technical tasks. On top of that, one still needs room for OS, software and KV cache. 8GB is just not very useful for local LLMs. That said, it can still be entertaining to try out a 4-bit 8B model for the fun of it.
[ Parent
| on HN ]
Re:Ggml.ai joins Hugging Face to ensure the long-term progre...
by zozbot234 on Saturday February 21, 2026 @07:18AM
(#47100090)
100B+ is the amount of total parameters, whereas what matters here is active - very different for sparse MoE models. You're right that there's some overhead for the OS/software stack but it's not that much. KV-cache is a good candidate for being swapped out, since it only gets a limited amount of writes per emitted token.
Re:Ggml.ai joins Hugging Face to ensure the long-term progre...
by zargon on Saturday February 21, 2026 @10:01AM
(#47101456)
Total parameters, not active parameters, is the property that matters for model robustness under extreme quantization.
Once you're swapping from disk, the performance will be quite unusable for most people. And for local inference, KV cache is the worst possible choice to put on disk.