Building a personal, private AI computer on a budget

Re:Building a personal, private AI computer on a budget by datadrivenangel on Tuesday February 11, 2025 @09:34AM (#43013113)

You pay a premium to get the theoretical local privacy and reliability of hosting your own models.

But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by joshstrange on Tuesday February 11, 2025 @09:44AM (#43013267)

Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other.

I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by nickthegreek on Tuesday February 11, 2025 @09:54AM (#43013409)

You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by joshstrange on Tuesday February 11, 2025 @01:34PM (#43016408)

Oh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by nickthegreek on Tuesday February 11, 2025 @02:39PM (#43017335)

I decided on openai api for now after setting up so many differnt methods. the local stuff isn't up to snuff yet for what I am trying to accomplish but decent for basic control.

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by joshstrange on Tuesday February 11, 2025 @02:59PM (#43017602)

I use a combo of Anthropic and OpenAI for now through my bots and my chat UIs and that lets me iterate faster. My hope is once I've done all my testing I could consider moving to local models if it made sense.

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by cruffle_duffle on Tuesday February 11, 2025 @10:37AM (#43014032)

> You don't know what you will do with it until you have it and it starts unlocking things in your mind

Exactly. Once the price and performance get to the level where buying stuff for local training and inferencing… that is when we will start to see the LLM break out of its current “corporate lawyer safe” stage and really begin to shake things up.

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by rsanek on Tuesday February 11, 2025 @09:42AM (#43013221)

With something like OpenRouter, you don't even have to manually integrate with multiple vendors

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by wkat4242 on Tuesday February 11, 2025 @05:28PM (#43019195)

Is that like LiteLLM? I have that running but never tried OpenRouter. I wonder now if it's better :)

[ Parent | on HN ]

Re:Building a personal, private AI computer on a budget by pelatimtt on Tuesday February 11, 2025 @07:39PM (#43020306)

You can also try LangDB or Portkey

[ Parent | on HN ]