Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by zozbot234 on Friday February 20, 2026 @09:40AM (#47088598)

> We still need good value hardware to run Kimi/GLM in-house

If you stream weights in from SSD storage and freely use swap to extend your KV cache it will be really slow (multiple seconds per token!) but run on basically anything. And that's still really good for stuff that can be computed overnight, perhaps even by batching many requests simultaneously. It gets progressively better as you add more compute, of course.

[ Parent | on HN ]

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by HPsquared on Friday February 20, 2026 @10:07AM (#47088923)

At a certain point the energy starts to cost more than renting some GPUs.

[ Parent | on HN ]

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by vardalab on Friday February 20, 2026 @12:48PM (#47091282)

Yeah, that is hard to argue with because I just go to OpenRouter and play around with a lot of models before I decide which ones I like. But there's something special about running it locally in your basement

[ Parent | on HN ]

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by dotancohen on Friday February 20, 2026 @08:09PM (#47096344)

I'd love to hear more about this. How do you decide that you like a model? For which use cases?

[ Parent | on HN ]

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by fc417fc802 on Friday February 20, 2026 @03:04PM (#47093165)

Aren't decent GPU boxes in excess of $5 per hour? At $0.20 per kWhr (which is on the high side in the US) running a 1 kW workstation 24/7 would work out to the same price as 1 hour of GPU time.

The issue you'll actually run into is that most residential housing isn't wired for more than ~2kW per room.

[ Parent | on HN ]

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by Aurornis on Friday February 20, 2026 @01:32PM (#47091837)

> it will be really slow (multiple seconds per token!)

This is fun for proving that it can be done, but that's 100X slower than hosted models and 1000X slower than GPT-Codex-Spark.

That's like going from real time conversation to e-mailing someone who only checks their inbox twice a day if you're lucky.

[ Parent | on HN ]

Re:Ggml.ai joins Hugging Face to ensure the long-term progre... by zozbot234 on Saturday February 21, 2026 @07:10AM (#47100040)

You'd need real rack-scale/datacenter infrastructure to properly match the hosted models that are keeping everything in fast VRAM at all times, and then you only get reasonable utilization on that by serving requests from many users. The ~100X slower tier is totally okay for experimentation and non-conversational use cases (including some that are more agentic-like!), and you'd reach ~10X (quite usable for conversation) by running something like a good homelab.

[ Parent | on HN ]