Gemma 3 QAT Models: Bringing AI to Consumer GPUs

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by kristianp on Monday April 21, 2025 @05:20PM (#43756621)

Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Tuesday April 22, 2025 @08:45AM (#43761457)

Thanks for your suggestion!

I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Tuesday April 22, 2025 @08:48AM (#43761485)

Now this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?

[ Parent | on HN ]