News for hackers. Stuff that matters.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by holografix on Sunday April 20, 2025 @09:27AM (#43743631)
Could 16gb vram be enough for the 27b QAT version?
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jffry on Sunday April 20, 2025 @09:39AM (#43743704)
With `ollama run gemma3:27b-it-qat "What is blue"`, GPU memory usage is just a hair over 20GB, so no, probably not without a nerfed context window
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by woadwarrior01 on Sunday April 20, 2025 @09:58AM (#43743804)
Indeed, the default context length in ollama is a mere 2048 tokens.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by hskalin on Sunday April 20, 2025 @10:02AM (#43743825)
With ollama you could offload a few layers to cpu if they don't fit in the VRAM. This will cost some performance ofcourse but it's much better than the alternative (everything on cpu)
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by senko on Sunday April 20, 2025 @12:09PM (#43744666)
I'm doing that with a 12GB card, ollama supports it out of the box.

For some reason, it only uses around 7GB of VRAM, probably due to how the layers are scheduled, maybe I could tweak something there, but didn't bother just for testing.

Obviously, perf depends on CPU, GPU and RAM, but on my machine (3060 + i5-13500) it's around 2 t/s.

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by dockerd on Monday April 21, 2025 @10:22AM (#43752342)
Does it work on LM Studio? Loading 27b-it-qat taking up more than 22GB on 24GB mac.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by halflings on Sunday April 20, 2025 @09:28AM (#43743634)
That's what the chart says yes. 14.1GB VRAM usage for the 27B model.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by erichocean on Sunday April 20, 2025 @09:34AM (#43743678)
That's the VRAM required just to load the model weights.

To actually use a model, you need a context window. Realistically, you'll want a 20GB GPU or larger, depending on how many tokens you need.

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by oezi on Sunday April 20, 2025 @10:05AM (#43743834)
I didn't realize that the context would require such so much memory. Is this KV caches? It would seem like a big advantage if this memory requirement could be reduced.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by parched99 on Sunday April 20, 2025 @11:06AM (#43744249)
I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp.

Prompt Tokens: 10

Time: 229.089 ms

Speed: 43.7 t/s

Generation Tokens: 41

Time: 959.412 ms

Speed: 42.7 t/s

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by floridianfisher on Sunday April 20, 2025 @03:20PM (#43745881)
Try one of the smaller versions. 27b is too big for your gpu
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by parched99 on Sunday April 20, 2025 @04:04PM (#43746177)
I'm aware. I was addressing the question being asked.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by tbocek on Sunday April 20, 2025 @03:40PM (#43746002)
This is probably due to this: https://github.com/ggml-org/llama.cpp/issues/12637. This GitHub issue is about interleaved sliding window attention (iSWA) not available in llama.cpp for Gemma 3. This could reduce the memory requirements a lot. They mentioned for a certain scenario, going from 62GB to 10GB.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by parched99 on Sunday April 20, 2025 @04:25PM (#43746296)
Resolving that issue, would help reduce (not eliminate) the size of the context. The model will still only just barely fit in 16 GB, which is what the parent comment asked.

Best to have two or more low-end, 16GB GPUs for a total of 32GB VRAM to run most of the better local models.

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by nolist_policy on Monday April 21, 2025 @04:22AM (#43749521)
Ollama supports iSWA.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by idonotknowwhy on Sunday April 20, 2025 @07:31PM (#43747323)
I didn't realise the 5070 is slower than the 3090. Thanks.

If you want a bit more context, try -ctv q8 -ctk q8 (from memory so look it up) to quant the kv cache.

Also an imatrix gguf like iq4xs might be smaller with better quality

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by parched99 on Sunday April 20, 2025 @09:50PM (#43747892)
I answered the question directly. IQ4_X_S is smaller, but slower and less accurate than Q4_0. The parent comment specifically asked about the QAT version. That's literally what this thread is about. The context-length mention was relevant to show how it's only barely usable.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by abawany on Monday April 21, 2025 @04:42PM (#43756253)
I tried the 27b-iat model on a 4090m with 16gb vram with mostly default args via llama.cpp and it didn't fit - used up the vram and tried to use about 2gb of system ram: performance in this setup was < 5 tps.