News for hackers. Stuff that matters.
Re:My local model setup on an M4 Pro Mac Mini by visarga on Wednesday September 02, 2026 @03:47AM (#49533100)
Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting.

If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

Re:My local model setup on an M4 Pro Mac Mini by BoredomIsFun on Wednesday September 02, 2026 @04:26AM (#49533415)
> not 1 token on 1 session like local models.

Local models can absolutely run in batch, what are even talking about?

> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.

Re:My local model setup on an M4 Pro Mac Mini by helsinkiandrew on Wednesday September 02, 2026 @06:38AM (#49534389)
> Local models can absolutely run in batch, what are even talking about?

I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.