News for hackers. Stuff that matters.

A comment on Gemma 3 QAT Models: Bringing AI to Consumer GPUs. Parent comment

Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by codybontecou on Sunday April 20, 2025 @05:55PM (#43746789)
Can you run the mlx-variation of this model through Ollama so that I can interact with it in Open WebUI?
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by simonw on Sunday April 20, 2025 @06:37PM (#43747028)
I haven't tried it yet but there's an MLX project that exposes an OpenAI-compatible serving endpoint that should work with Open WebUI: https://github.com/madroidmaq/mlx-omni-server
[ Parent | on HN ]
Re:Gemma 3 QAT Models: Bringing AI to Consumer GPUs by codybontecou on Monday April 21, 2025 @09:49AM (#43751993)
Appreciate the link. I'll try to tinker with it later today.
[ Parent | on HN ]