News for hackers. Stuff that matters.
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by jffry on Sunday April 20, 2025 @09:39AM (#43743704)
With `ollama run gemma3:27b-it-qat "What is blue"`, GPU memory usage is just a hair over 20GB, so no, probably not without a nerfed context window
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by woadwarrior01 on Sunday April 20, 2025 @09:58AM (#43743804)
Indeed, the default context length in ollama is a mere 2048 tokens.