News for hackers. Stuff that matters.

parched99

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs on Sunday April 20, 2025 @09:50PM (#43747892)
I answered the question directly. IQ4_X_S is smaller, but slower and less accurate than Q4_0. The parent comment specifically asked about the QAT version. That's literally what this thread is about. The context-length mention was relevant to show how it's only barely usable.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs on Sunday April 20, 2025 @04:25PM (#43746296)
Resolving that issue, would help reduce (not eliminate) the size of the context. The model will still only just barely fit in 16 GB, which is what the parent comment asked.

Best to have two or more low-end, 16GB GPUs for a total of 32GB VRAM to run most of the better local models.

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs on Sunday April 20, 2025 @04:04PM (#43746177)
I'm aware. I was addressing the question being asked.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs on Sunday April 20, 2025 @11:52AM (#43744568)
I think Powershell is a bad test. I've noticed all local models have trouble providing accurate responses to Powershell-related prompts. Strangely, even Microsoft's model, Phi 4, is bad at answering these questions without careful prompting. Though, MS can't even provide accurate PS docs.

My best guess is that there's not enough discussion/development related to Powershell in training data.

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs on Sunday April 20, 2025 @11:06AM (#43744249)
I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp.

Prompt Tokens: 10

Time: 229.089 ms

Speed: 43.7 t/s

Generation Tokens: 41

Time: 959.412 ms

Speed: 42.7 t/s