Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs
by abawany on Monday April 21, 2025 @04:42PM
(#43756253)
I tried the 27b-iat model on a 4090m with 16gb vram with mostly default args via llama.cpp and it didn't fit - used up the vram and tried to use about 2gb of system ram: performance in this setup was < 5 tps.
[ Parent
| on HN ]
There are no comments on this comment yet.