News for hackers. Stuff that matters.
Re:My local model setup on an M4 Pro Mac Mini by BoredomIsFun on Wednesday September 02, 2026 @04:26AM (#49533415)
> not 1 token on 1 session like local models.

Local models can absolutely run in batch, what are even talking about?

> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.

Re:My local model setup on an M4 Pro Mac Mini by helsinkiandrew on Wednesday September 02, 2026 @06:38AM (#49534389)
> Local models can absolutely run in batch, what are even talking about?

I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.