Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs
by jffry on Sunday April 20, 2025 @09:39AM
(#43743704)
With `ollama run gemma3:27b-it-qat "What is blue"`, GPU memory usage is just a hair over 20GB, so no, probably not without a nerfed context window
[ Parent
| on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs
by woadwarrior01 on Sunday April 20, 2025 @09:58AM
(#43743804)
Indeed, the default context length in ollama is a mere 2048 tokens.