News for hackers. Stuff that matters.

A comment on Gemma 3 QAT Models: Bringing AI to Consumer GPUs. Parent comment

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by simonw on Sunday April 20, 2025 @11:47AM (#43744532)
Interestingly the Gemma 3 docs say: https://ai.google.dev/gemma/docs/core/model_card_3#:~:text=T...

> Total output context up to 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size per request, subtracting the request input tokens

I don't know how to get it to output anything that length though.

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by nico on Sunday April 20, 2025 @12:14PM (#43744700)
Thank you for the insights and useful links

Will keep experimenting, will also try mistral3.1

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by Gracana on Sunday April 20, 2025 @03:43PM (#43746033)
You can probably find some long-form tuned models on HF. I've had decent results with QwQ-32B (which I can run on my desktop) and Mistral Large (which I have to run on my server). Generating and refining an outline before writing the whole piece can help, and you can also split the piece up into multiple outputs (working a paragraph or two at a time, for instance). So far I've found it to be a tough process, with mixed results.
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by nico on Sunday April 20, 2025 @05:28PM (#43746653)
Thank you, will try out your suggestions

Have you used something like a director model to supervise the output? If so, could you comment on the effectiveness of it and potentially any tips?

[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by Gracana on Sunday April 20, 2025 @08:27PM (#43747588)
Nope, sounds neat though. There's so much to keep up with in this space.
[ Parent | on HN ]