News for hackers. Stuff that matters.
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by dulakian on Wednesday March 05, 2025 @07:59PM (#43274869)
I am using the Q6_K_L quant and it's running at about 40G of vram with the KV cache.

Device 1 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||20.170Gi/23.988Gi]

Device 2 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||19.945Gi/23.988Gi]

Re:QwQ-32B: Embracing the Power of Reinforcement Learning by lostmsu on Thursday March 06, 2025 @12:02AM (#43276600)
What's the context length?
Re:QwQ-32B: Embracing the Power of Reinforcement Learning by dulakian on Thursday March 06, 2025 @10:16AM (#43281183)
The model has a context of 131,072, but I only have 48G of VRAM so I run it with a context of 32768.