Gemma 3 QAT Models: Bringing AI to Consumer GPUs

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Tuesday April 22, 2025 @08:45AM (#43761457)

Thanks for your suggestion!

I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by pantulis on Tuesday April 22, 2025 @08:48AM (#43761485)

Now this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?

[ Parent | on HN ]