News for hackers. Stuff that matters.

A comment on Gemma 3 QAT Models: Bringing AI to Consumer GPUs. Parent comment

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jonaustin on Monday April 21, 2025 @10:29PM (#43758570)
On a M4 Max 128GB via LM Studio:

query: "make me a snake game in python with pygame"

(mlx 4 bit quant) mlx-community/gemma-3-27b-it-qat@4bit: 26.39 tok/sec • 1681 tokens 0.63s to first token

(gguf 4 bit quant) lmstudio-community/gemma-3-27b-it-qat: 22.72 tok/sec • 1866 tokens 0.49s to first token

using Unsloth's settings: https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-...

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by yencabulator on Tuesday April 22, 2025 @03:04PM (#43765225)
I genuinely would have expected a $3,500+ setup to do better than just 10x pure-CPU on a AMD Ryzen 9 8945HS.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jonaustin on Saturday April 26, 2025 @03:01AM (#43801505)
Find another laptop that does that well.
[ Parent | on HN ]