Gemma 3 QAT Models: Bringing AI to Consumer GPUs

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Monday April 21, 2025 @07:21AM (#43750648)

> Can you quote tps?

LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jychang on Monday April 21, 2025 @07:37AM (#43750755)

That's pretty terrible. I'm getting 18tok/sec Gemma 3 27b QAT on a M1 Max 32gb macbook.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Monday April 21, 2025 @07:45AM (#43750823)

Yeah, I know. Not sure if this due to something in LLM Studio or whatever.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Saturday April 26, 2025 @01:19PM (#43805417)

I was running the mac in low power mode!!! Getting 24tps now.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by kristianp on Monday April 21, 2025 @05:20PM (#43756621)

Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Tuesday April 22, 2025 @08:45AM (#43761457)

Thanks for your suggestion!

I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Tuesday April 22, 2025 @08:48AM (#43761485)

Now this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Saturday April 26, 2025 @01:18PM (#43805414)

Gah, turns out I was running the Mac in low power mode!

I get 24tps in LM Studio now with gemma-3-27b-it-qat.

[ Parent | on HN ]