Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs
by oezi on Sunday April 20, 2025 @10:05AM
(#43743834)
I didn't realize that the context would require such so much memory. Is this KV caches? It would seem like a big advantage if this memory requirement could be reduced.
To actually use a model, you need a context window. Realistically, you'll want a 20GB GPU or larger, depending on how many tokens you need.