Re:QwQ-32B: Embracing the Power of Reinforcement Learning
by zamalek on Wednesday March 05, 2025 @07:23PM
(#43274616)
39gb if you use a fp8 quantized model.[1] Remember that your OS might be using some of that itself.
As far as I recall, Ollama/llama.cpp recently added a feature to page-in parameters - so you'll be able to go arbitrarily large soon enough (at a performance cost). Obviously more in RAM = more speed = more better.
As far as I recall, Ollama/llama.cpp recently added a feature to page-in parameters - so you'll be able to go arbitrarily large soon enough (at a performance cost). Obviously more in RAM = more speed = more better.
[1]: https://token-calculator.net/llm-memory-calculator