News for hackers. Stuff that matters.
Re:Experimenting with Local LLMs on macOS by JumpCrisscross on Monday September 08, 2025 @12:17PM (#45170155)
> Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible

Oh wow, a maxed out Studio could run a 600B parameter model entirely in memory. Not bad for $12k.

There may be a business in creating the software that links that box to an app on your phone.

Re:Experimenting with Local LLMs on macOS by brokencode on Monday September 08, 2025 @01:54PM (#45171529)
Perhaps said software could even form an end to end encrypted tunnel from your phone to your local LLM server anywhere over the internet via a simple server intermediary.

The amount of data transferred is tiny and the latency costs are typically going to be dominated by the LLM inference anyway. Not much advantage to doing LAN only except that you don’t need a server.

Though the amount of people who care enough to buy a $3k - $10k server and set this up compared to just using ChatGPT is probably very small.

Re:Experimenting with Local LLMs on macOS by JumpCrisscross on Monday September 08, 2025 @01:57PM (#45171567)
> amount of people who care enough to buy a $3k - $10k server and set this up compared to just using ChatGPT is probably very small

So I maxed that out, and it’s with Apple’s margins. I suspect you could do it for $5k.

I’d also note that for heavy users of ChatGPT, the difference in energy costs for a home setup and the price for ChatGPT tokens may make this financially compelling for heavy users.

Re:Experimenting with Local LLMs on macOS by brokencode on Monday September 08, 2025 @02:58PM (#45172344)
True, it may be profitable for pro users. At $200 a month for ChatGPT Pro, it may only take a few years to recoup the initial costs. Not sure about energy costs though.

And of course you’d be getting a worse model, since no open source model currently is as good as the best proprietary ones.

Though that gap should narrow as the open models improve and the proprietary ones seemingly plateau.

Re:Experimenting with Local LLMs on macOS by dghlsakjg on Monday September 08, 2025 @02:20PM (#45171871)
That software is an HTTP request, no?

Any number of AI apps allow you to specify a custom endpoint. As long as your AI server accepts connections to the internet, you're gravy.

Re:Experimenting with Local LLMs on macOS by JumpCrisscross on Monday September 08, 2025 @04:10PM (#45173310)
> That software is an HTTP request, no?

You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.

Re:Experimenting with Local LLMs on macOS by dghlsakjg on Monday September 08, 2025 @05:51PM (#45174498)
My point is, what is there to build?

The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain.

Everyone else will continue to just use app based subscriptions.

Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare.

Why would AI buck the trend that every other computing service has followed?

Re:Experimenting with Local LLMs on macOS by JumpCrisscross on Monday September 08, 2025 @06:15PM (#45174745)
> what is there to build?

Integrated solution. You buy the box. You download the app. It works like the ChatGPT app, except it's tunneling to the box you have at home which has been preconfigured to work with the app. Maybe you have a subscription to keep everything up to date. Maybe you have an open-source model 'store'.

Re:Experimenting with Local LLMs on macOS by itsn0tm3 on Monday September 08, 2025 @11:09PM (#45176946)
You don’t tell your media player company secrets ;)

I think there is a market here, solely based on actual data privacy. Not sure how big it is but I can see quite some companies have use for it.

Re:Experimenting with Local LLMs on macOS by dghlsakjg on Monday September 08, 2025 @11:33PM (#45177097)
> You don’t tell your media player company secrets ;)

No, but my email provider has a de-facto repository of incredibly sensitive documents. When you put convenience and cost up against privacy, the market has proven over and over that no one gives a shit.

Re:Experimenting with Local LLMs on macOS by simonw on Monday September 08, 2025 @06:37PM (#45175004)
I have been using a Tailscale VPN to make LM Studio and Ollama running on my Mac available to my iPhone when I leave the house.