> but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast.

Same, esp. if you factor in the cost of renting. Even if you run 24/7 it's hard to see it paying off in half the time it will take to be obsolete

You pay a premium to get the theoretical local privacy and reliability of hosting your own models.

But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.

Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other.

I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.

You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.

Oh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).

I decided on openai api for now after setting up so many differnt methods. the local stuff isn't up to snuff yet for what I am trying to accomplish but decent for basic control.

I use a combo of Anthropic and OpenAI for now through my bots and my chat UIs and that lets me iterate faster. My hope is once I've done all my testing I could consider moving to local models if it made sense.

> You don't know what you will do with it until you have it and it starts unlocking things in your mind

Exactly. Once the price and performance get to the level where buying stuff for local training and inferencing… that is when we will start to see the LLM break out of its current “corporate lawyer safe” stage and really begin to shake things up.

With something like OpenRouter, you don't even have to manually integrate with multiple vendors

Is that like LiteLLM? I have that running but never tried OpenRouter. I wonder now if it's better :)

You can also try LangDB or Portkey

For what it's worth, looking at the benchmarks, I think the machine they built is comparable to what your MBP can already do. They probably have a better inference speed, though.

A Strix Halo minipc might be a good mid tier option once they're out, though AMD still isn't clear on how much they'll overprice them.

Core Ultra Arc iGPU boxes are pretty neat too for being standalone and can be loaded up with DDR5 shared memory, efficient and usable in terms of speed, though that's definitely low end performance, plus SYCL and IPEX are a bit eh.

The juice aint worth the squeeze to do this locally.

But you should still play with proxmox, just not for this purpose. My recommendation would be to get an i7 HP Elitedesk. I have multiple racks in my basement, hundreds of gigs of ram, multiple 2U 2x processor enterprise servers etc.... but at this point all of it is turned off and a single HP Elitedesk with a 2nd NIC added and 64GB of ram is doing everything I ever needed and more.

Yeah, right now I'm running a tower PC (Intel Core i9-11900K, 64GB Ram) with Unraid as my local "app server". I want to play with Proxmox (for professional and mostly fun reasons) though. Someday I'd like a rack in my basement as my homelab stuff has overgrown the space it's in and I'm going to need to add a new 12-bay Synology (on top of 2x12-bay) soon since I'm running out of space again. For now I've been sticking with consumer/prosumer equipment but my needs are slowly outstripping that I think.