News for hackers. Stuff that matters.

A comment on Gemma 3 QAT Models: Bringing AI to Consumer GPUs. Parent comment

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jychang on Monday April 21, 2025 @03:44AM (#43749324)
MLX is slower than GGUFs on Macs.

On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by phaedrix on Monday April 21, 2025 @11:12AM (#43752864)
No, in general mlx versions are always faster, ice tested most of them.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by 85392_school on Monday April 21, 2025 @11:44AM (#43753241)
What TPS difference are you getting?
[ Parent | on HN ]