News for hackers. Stuff that matters.

A comment on Gemma 3 QAT Models: Bringing AI to Consumer GPUs. Parent comment

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by simonw on Sunday April 20, 2025 @11:07AM (#43744258)
MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by Elucalidavah on Sunday April 20, 2025 @01:01PM (#43744971)
> MacBook Pro M2 with 64GB of RAM

Are there non-mac options with similar capabilities?

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by simonw on Sunday April 20, 2025 @01:13PM (#43745043)
Yes, but I don't really know anything about those. https://www.reddit.com/r/LocalLLaMA/ is full of people running models on PCs with NVIDIA cards.

The unique benefit of an Apple Silicon Mac at the moment is that the 64GB of RAM is available to both the GPU and the CPU at once. With other hardware you usually need dedicated separate VRAM for the GPU.

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by _neil on Sunday April 20, 2025 @02:55PM (#43745718)
It’s not out yet, but the upcoming Framework desktop [0] is supposed to have a similar unified memory setup.

[0] https://frame.work/desktop

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by dwood_dev on Sunday April 20, 2025 @04:49PM (#43746431)
Anything with the Radeon 8060S/Ryzen AI Max+ 395. One of the popular MiniPC Chinese brands has them for preorder[0] with shipping starting May 7th. Framework also has them, but shipping Q3.

0: https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-a...

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by chpatrick on Sunday April 20, 2025 @09:22PM (#43747791)
I've never been able to get ROCm working reliably personally.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by danans on Sunday April 20, 2025 @07:34PM (#43747338)
Nvidia Orin AGX if a desktop form factor works for you.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by chpatrick on Sunday April 20, 2025 @09:21PM (#43747789)
I remember seeing a post about someone running the full size DeepSeek model in a dual-Xeon server with a ton of RAM.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jychang on Monday April 21, 2025 @03:44AM (#43749324)
MLX is slower than GGUFs on Macs.

On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by phaedrix on Monday April 21, 2025 @11:12AM (#43752864)
No, in general mlx versions are always faster, ice tested most of them.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by 85392_school on Monday April 21, 2025 @11:44AM (#43753241)
What TPS difference are you getting?
[ Parent | on HN ]