News for hackers. Stuff that matters.

A comment on Qwen3-235B-A22B-Thinking-2507. Parent comment

Re:Qwen3-235B-A22B-Thinking-2507 by aliljet on Friday July 25, 2025 @11:31AM (#44684291)
I see the term 'local inference' everywhere. It's an absurd misnomer without hardware and cost defined. I can also run a coal fired power plant in my backyard, but in practice, there's no reasonable way to make that economical beyond being a toy.

(And I should add, you are a hero for doing this work, only love in my comment, but still a demand for detail$!)

[ Parent | on HN ]
Re:Qwen3-235B-A22B-Thinking-2507 by regularfry on Friday July 25, 2025 @11:53AM (#44684600)
Hardware and cost is assumed to be approximately desktop-class. If you've got a gaming rig with an RTX 4090 and 128MB RAM, you can run this if you pick the right quant.
[ Parent | on HN ]
Re:Qwen3-235B-A22B-Thinking-2507 by cmpxchg8b on Friday July 25, 2025 @11:58AM (#44684670)
128MB? Quantization has come a long way!
[ Parent | on HN ]
Re:Qwen3-235B-A22B-Thinking-2507 by danielhanchen on Friday July 25, 2025 @04:20PM (#44687959)
I think they mis-spoke 128GB* :)
[ Parent | on HN ]
Re:Qwen3-235B-A22B-Thinking-2507 by regularfry on Friday July 25, 2025 @05:05PM (#44688460)
Wishful thinking there on my part.
[ Parent | on HN ]
Re:Qwen3-235B-A22B-Thinking-2507 by danielhanchen on Friday July 25, 2025 @05:38PM (#44688794)
Though technically < 1GB is enough - one had offload it to the SSD, albeit with very slow speeds!
[ Parent | on HN ]
Re:Qwen3-235B-A22B-Thinking-2507 by danielhanchen on Friday July 25, 2025 @04:24PM (#44687979)
The trick of llama.cpp and our dynamic quants is you can actually offload the model to RAM / even an SSD! If you have GPU VRAM + RAM + SSD > the model size (say 90GB for dynamic 2bit quant), then it'll run well!

Ie you can actually run it on a local desktop or even your laptop now! You don't need a 90GB GPU for example, but say a 24GB GPU + 64GB to 128GB RAM.

The speeds are around 3 to 5 tokens / second, so still ok! I write more about improving speed for local devices here: https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tun...

[ Parent | on HN ]