Like I said, seems HPC is lost art. All four sub replies missed that in my original comment.
I agree with you, and i would like to set up a server at work for local inference but we simply do not have the knowledge, it's much easier to get a couple of AI Max machines that are good enough for me and the boss which i convinced on the good points on running clankers locally and be done with it (though my plan is to use the AI MAX machines to figure out what the infrastructure and workflow will look like, and at some point ask for the investment)
damn, too bad that the laptops with the 470 are capped at 64GB
How is this literally even possible in 2026? In a thread about LLMs, no less?
Then there's testing the approaches and see whatever sticks best.
Not saying they're right.
For individuals though, it’s the same reason plenty of people pay for the 2tb or 4tb SSD for their laptops or desktops even though NAS has been a thing for 40 years. Maintaining a “server” of any sort is a completely separate endeavor for most. Just like a NAS is a no-brainer for many and non-starter for many more.
The problem with LLMs outside of server is that people typically have many devices. So you can run LLM on your laptop. You want to make a query from your phone. Start your laptop? You want to do something from your smart home? Start your laptop? What about your partner? Will they need to buy another top of the line laptop for themselves? Your kids?
Alternative to server is of course that everything will have over 100 GB of memory, eat battery like crazy, be dumber and slower than single server in your household.
4 TB ssd was like $300 until recently. Even if RAM prices were to go to previous values, I don't think AI capability will be any comparable. Besides you also need proper compute, unless you want your prefill to be snail slow.
Every scenario you described applies the same to a NAS. What if you need a file from your phone? Open your laptop? What if your partner needs a file? Open your laptop? The answer to all of them is either yes, or they have it on their laptop/phone too.
Apple (and other laptop makers) weren’t charging $300 for 4tb even before the shortage. They realized that not everyone needed that and they way overcharged for that but still less than today’s prices. People who needed it bought it even though it would have been a lot cheaper for them to build a NAS. A current-gen M5 is perfectly capable of doing decent tk/s given enough RAM. In 2 or 3 years it would be even more capable, it just needs RAM.
Plenty of people will use clou hosted models like they do with storage today. Plenty of people will build home servers for AI. Just like plenty of people have home NASs. But the number of people who just want a large SSD on their laptop make the previous group a rounding error and those people would rather have a beefy laptop or desktop with enough RAM in it. If anyone can solve the Phone<->Desktop communication seamlessly it’s Apple after all.
Unlike storage however, a lot of people are now aware of how much data collection tech and AI companies are doing and the idea that AI models are training on their personal photos and prompts is deeply unappealing to many. With just storage, what could Google or Apple do? Certainly they wouldn’t just print your photos and put them in an ad or on their website. But with AI models, it’s a different equation
No. Files are your property. Inference is a service.
There's also the fact that working with files over network is often ill-advised and borderline impossible for some workloads. Even when that NAS is connected over wire and not wireless. Meanwhile even if you're on so-so connection, LLMs work great.
NAS is mostly good for backups, torrenting linux isos and serving them with Plex or Jellyfin. You are not installing software on your NAS instead of your machine. Or doing video editing. Or playing demanding games. I have file server at home and I still have ~8 TB of storage in my desktop, of which at leat 3 TB I can't move out.
B. If I have 1 TB of my work files, my wife has 1 TB of her work files, that's 2 TB of total storage. If both us need LLM, it doesn't mean we need twice the LLM as one of us.
And with a device like this, you pay one single time to buy the product and then never have to think about paying for tokens ever again. For ordinary users that's silly, but for users or businesses chewing through lots of tokens, this means they can create their own turnkey replacement for "the cloud" where the only cost after purchasing the machine is the electricity to run it (which with these machines is very little). Mix that with the privacy benefits and suddenly it makes a lot of sense.