Re:Accelerating Gemma 4: faster inference with multi-token p...
by aimxhaisse on Tuesday May 05, 2026 @04:03PM
(#48027728)
It even fits on a 3060 with turboquant / Q4 at decent speed (40T/s) for ~200$ (:
Re:Accelerating Gemma 4: faster inference with multi-token p...
by 2ndorderthought on Tuesday May 05, 2026 @03:20PM
(#48027222)
Some of the early quants for qwen3.6 were broken. It's still finicky but with a little hand holding it's crazy.
Local models are the future it's awesome
Re:Accelerating Gemma 4: faster inference with multi-token p...
by jszymborski on Tuesday May 05, 2026 @01:34PM
(#48025782)
The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.
Re:Accelerating Gemma 4: faster inference with multi-token p...
by maille on Tuesday May 05, 2026 @06:19PM
(#48029500)
Bad at coding, but would it be good at code review?
Re:Accelerating Gemma 4: faster inference with multi-token p...
by avadodin on Wednesday May 06, 2026 @05:47AM
(#48034298)
Good compared to what?
Nothing? Probably better.
Re:Accelerating Gemma 4: faster inference with multi-token p...
by moffkalast on Tuesday May 05, 2026 @04:55PM
(#48028426)
The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.
I tried first with Qwen but it was unstable and had ridiculously long thinning traces!