News for hackers. Stuff that matters.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by manmal on Wednesday March 05, 2025 @05:20PM (#43273487)
I guess I won’t be needing that 512GB M3 Ultra after all.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by UncleOxidant on Wednesday March 05, 2025 @06:24PM (#43274198)
I think the Framework AI PC will run this quite nicely.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by Tepix on Thursday March 06, 2025 @08:29AM (#43279965)
I think you want a lot of speed to make up for the fact that it's so chatty. Two 24GB GPUs (so you have room for context) will probably be great.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by rpastuszak on Wednesday March 05, 2025 @06:54PM (#43274440)
How much vram do you need to run this model? Is 48 gb unified memory enough?
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by dulakian on Wednesday March 05, 2025 @07:59PM (#43274869)
I am using the Q6_K_L quant and it's running at about 40G of vram with the KV cache.

Device 1 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||20.170Gi/23.988Gi]

Device 2 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||19.945Gi/23.988Gi]

Re:QwQ-32B: Embracing the Power of Reinforcement Learning by lostmsu on Thursday March 06, 2025 @12:02AM (#43276600)
What's the context length?
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by dulakian on Thursday March 06, 2025 @10:16AM (#43281183)
The model has a context of 131,072, but I only have 48G of VRAM so I run it with a context of 32768.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by zamalek on Wednesday March 05, 2025 @07:23PM (#43274616)
39gb if you use a fp8 quantized model.[1] Remember that your OS might be using some of that itself.

As far as I recall, Ollama/llama.cpp recently added a feature to page-in parameters - so you'll be able to go arbitrarily large soon enough (at a performance cost). Obviously more in RAM = more speed = more better.

[1]: https://token-calculator.net/llm-memory-calculator

Re:QwQ-32B: Embracing the Power of Reinforcement Learning by daemonologist on Wednesday March 05, 2025 @07:21PM (#43274598)
The quantized model fits in about 20 GB, so 32 would probably be sufficient unless you want to use the full context length (long inputs and/or lots of reasoning). 48 should be plenty.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by manmal on Thursday March 06, 2025 @01:17AM (#43276971)
I‘ve tried the very early Q4 mlx release on an M1 Max 32GB (LM Studio @ default settings), and have run into severe issues. For the coding tasks I gave it, it froze before it was done with reasoning. I guess I should limit context size. I do love what I‘m seeing though, the output reads very similar to R1, and I mostly agree with its conclusions. The Q8 version has to be way better even.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by whitehexagon on Thursday March 06, 2025 @11:32AM (#43282063)
Does the Q8 fit within your 32GB (also using an M1 32GB)
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by manmal on Thursday March 06, 2025 @12:30PM (#43282811)
No, Q4 just barely fits, and with a longer context sometimes things freeze. I definitely have to close Xcode.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by brandall10 on Wednesday March 05, 2025 @08:56PM (#43275382)
It's enough for 6 bit quant with a somewhat restricted context length.

Though based on the responses here, it needs sizable context to work, so we may be limited to 4 bit (I'm on an M3 Max w/ 48gb as well).

Re:QwQ-32B: Embracing the Power of Reinforcement Learning by seanmcdirmid on Thursday March 06, 2025 @01:28PM (#43283416)
A max with 64 GB of ram should be able to run this (I hope). I have to wait until an MLX model is available to really evaluate its speed, though.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by mettamage on Thursday March 06, 2025 @02:03PM (#43283915)
Yep, it does that. I have 64 GB and was actually running 40 GB of other stuff.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by pickettd on Thursday March 06, 2025 @08:58PM (#43286855)
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by seanmcdirmid on Friday March 07, 2025 @12:36PM (#43292182)
I downloaded the 8 bit quant last night but haven't had a chance to play with it yet.