Gemma 3 QAT Models: Bringing AI to Consumer GPUs

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by Gracana on Sunday April 20, 2025 @03:43PM (#43746033)

You can probably find some long-form tuned models on HF. I've had decent results with QwQ-32B (which I can run on my desktop) and Mistral Large (which I have to run on my server). Generating and refining an outline before writing the whole piece can help, and you can also split the piece up into multiple outputs (working a paragraph or two at a time, for instance). So far I've found it to be a tough process, with mixed results.

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by nico on Sunday April 20, 2025 @05:28PM (#43746653)

Thank you, will try out your suggestions

Have you used something like a director model to supervise the output? If so, could you comment on the effectiveness of it and potentially any tips?

[ Parent | on HN ]

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by Gracana on Sunday April 20, 2025 @08:27PM (#43747588)

Nope, sounds neat though. There's so much to keep up with in this space.

[ Parent | on HN ]