Re:Gemma 4 QAT models: Optimizing compression for mobile and...
by dofm on Friday June 05, 2026 @03:14PM
(#48416911)
This is safetensors. Is there any way to run these on a Mac paired with the MLX QAT?
(Pardon my ignorance; this stuff moves so fast)
Re:Gemma 4 QAT models: Optimizing compression for mobile and...
by thangalin on Friday June 05, 2026 @06:06PM
(#48418970)
Did you see this?
https://point.free/blog/gemma-4-on-a-2016-xeon/
Xeon, but could be useful for MTP on Mac.
Re:Gemma 4 QAT models: Optimizing compression for mobile and...
by dofm on Friday June 05, 2026 @06:15PM
(#48419044)
I hadn't seen this, thanks.
I do have the Qwen 3.6 (35B) MTP implementation running (in LM Studio; it doesn't need a separate drafter), along with non-MTP Gemma 4 26B, and I can see that Unsloth Studio can run the new QAT, but I can't see how you can run the assistant/drafter. Yet.
It's just a constantly changing landscape. Don't get me wrong, it's fascinating and for various reasons I am pleased I can keep up even slightly, but eeeehhh :-)
Re:Gemma 4 QAT models: Optimizing compression for mobile and...
by dofm on Monday June 08, 2026 @01:17AM
(#48441560)
To briefly follow up, as of yesterday llama.cpp can do Gemma 4's MTP, so I have this working at least initially — details here:
Re:Gemma 4 QAT models: Optimizing compression for mobile and...
by dofm on Saturday June 06, 2026 @02:01AM
(#48421856)
Yeah — that is the base QAT model, and there are safetensors weights for the QAT version of the MTP drafter, but there are no MLX/GGUF versions. I think the answer is a combination of:
1) Gemma 4 MTP is too fresh for off-the-shelf software to use anyway
2) "you can convert them yourself" which is fine, obvs
https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-un...