News for hackers. Stuff that matters.
Re:My local model setup on an M4 Pro Mac Mini by usrnm on Wednesday September 02, 2026 @03:31AM (#49532966)
> It's free

It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.

Re:My local model setup on an M4 Pro Mac Mini by visarga on Wednesday September 02, 2026 @03:45AM (#49533090)
If you buy the computer specifically for inference it is more expensive than cloud, but if you had it anyway it's free.
Re:My local model setup on an M4 Pro Mac Mini by trainingonme on Wednesday September 02, 2026 @03:52AM (#49533147)
True, but how many people (realistically) buy a computer with 48GB+ of RAM?
Re:My local model setup on an M4 Pro Mac Mini by NamlchakKhandro on Wednesday September 02, 2026 @04:04AM (#49533249)
48gb of vram.

a machine like this is about a years rent for most people.

a small car for most others.

Re:My local model setup on an M4 Pro Mac Mini by LeBit on Wednesday September 02, 2026 @04:48AM (#49533583)
I think he’s talking about the Mac Mini unified memory.

48G RAM is pretty useful if you want to run k8s locally for tests / exploration

Re:My local model setup on an M4 Pro Mac Mini by bel8 on Wednesday September 02, 2026 @07:14AM (#49534631)
true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires.

you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.

Re:My local model setup on an M4 Pro Mac Mini by LeBit on Wednesday September 02, 2026 @08:34AM (#49535320)
I actually have a Mac Mini M4 Pro with 48G. I gave the k8s example because this is what I was doing with it.

Was because I am back to using Linux as my workstation.

My Mac Mini is now a headless server for llama.cpp.

So, you are right that for these workloads , I would not be using the Mac Mini for k8s AND llama.

Another thing going against using a Mac for Linux containers is that there are no solutions that I know that properly manages memory : memory is given to the Linux vm , but never fluctuates if the needs in the vm are less than the initial request.

I know Orb Stack does that but is it proprietary. I think UTM does it , but not sure I would use UTM instead of Lima, Colima , multipass , etc to run containers.

Re:My local model setup on an M4 Pro Mac Mini by chrisweekly on Wednesday September 02, 2026 @08:41AM (#49535385)
Try smolvm microvms from https://smolmachines.com - among other benefits they only consume host resources if they're actually used.
Re:My local model setup on an M4 Pro Mac Mini by BoredomIsFun on Wednesday September 02, 2026 @03:42AM (#49533063)
> It's very possible that it costs you more than a cloud mode would

...which is almost always true in a single request/reply mode and never true in batch mode. Single request usually 2x-3x more expensive than cloud and batch mode 2x-3x cheaper. Now, for narrow tasks, a finetuned tiny 8b model would dramatically outperform SOTA frontiers for a fraction of price, esp. on energy efficient hardware like Apple.

Re:My local model setup on an M4 Pro Mac Mini by visarga on Wednesday September 02, 2026 @03:47AM (#49533100)
Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting.

If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

Re:My local model setup on an M4 Pro Mac Mini by BoredomIsFun on Wednesday September 02, 2026 @04:26AM (#49533415)
> not 1 token on 1 session like local models.

Local models can absolutely run in batch, what are even talking about?

> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.

Re:My local model setup on an M4 Pro Mac Mini by helsinkiandrew on Wednesday September 02, 2026 @06:38AM (#49534389)
> Local models can absolutely run in batch, what are even talking about?

I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.

Re:My local model setup on an M4 Pro Mac Mini by thrw93747572007 on Wednesday September 02, 2026 @07:58AM (#49534973)
It sounds like they are doing something similar to what I described in my other post below. Personal media station.

That can be done on hardware that quite a lot of people basically just have and don't use 24/7 to the max - because it is their gaming machine or their programming and compiling workhorse, for example. Of course you are paying for additional electricity but even with napkin-math instead of a "proper" calculation, you are unlikely to pay more for running your own instead of something commercial (and that can be offset further with some of the "modern" electricity contracts and/or PV and battery storage). Especially if we are talking about a stack that runs most of/all the time when you are not using your machine and makes LLM calls regularly while running.

The work in software/admin to get whatever you want set up is similiar no matter which infrastructure you use.

Re:My local model setup on an M4 Pro Mac Mini by albrewer on Thursday September 03, 2026 @11:32AM (#49551697)
> the cost is included in your electricity bill

Acting like an extra $20 on my electric bill is equivalent to a $200/mo subscription is... a take.

Re:My local model setup on an M4 Pro Mac Mini by exe34 on Thursday September 03, 2026 @11:59AM (#49552203)
Free heating in winter.
Re:My local model setup on an M4 Pro Mac Mini by a96 on Friday September 04, 2026 @12:02PM (#49566501)
Though much less energy efficient than a heat pump.
Re:My local model setup on an M4 Pro Mac Mini by exe34 on Saturday September 05, 2026 @04:00AM (#49574214)
Expensive heating but free computation then!