We need one of those specialized inference chip startups to succeed and a PC manufacturer willing to bet on them against Nvidia for the local AI to find mass market appeal.
For Immich, the cheapest option will either be a NAS or a used laptop depending on the amount of data you need, I wouldn't buy a mac for that.
(I just run the defaults on my CPU, works for me)
I think the decision comes primarily on how much data you would like to store for Immich, if you want to go cheaply, a 100 bucks used laptop will do the job, if you have too much data, a NAS will be more suitable (and you are certainly not going to get a mac where you can plug multiple internal hard drives for the price of a NAS)
That needs to be in (v)ram for searches.
Apple has always been the most cost effective choice for the value you get going all the way back to the Apple II, it's just that the floor of that cost has always been high. Anyone who thinks otherwise is a just a fanboy one way or the other.
That's how much many developers currently spend on tokens - every day. Whatever "Apple Tax" applies to a device that can run a capable model offline will amortise itself in a blink.
Current high-end Mac Studio with 32-core M3 Ultra chip and 96 GB of memory is $6800, 96GB is not enough to run GLM 5.2 without extreme quantization or stacking HW; but for the sake of discussion let's run quantized version on a single high end Mac Studio.
GLM 5.2 Max plan costs $ 112/m, so it would take ~60 months to recover the costs assuming the machine was bought just for AI. By then the current AI landscape would have changed drastically.
I use local AI on both Linux and Mac every single day, there's freedom, privacy and peace of mind in running the model locally. But I feel cost/value of local AI is overblown.
1. When Apple came out with the real Macbook Air in 2010/2011 (not the silly 2008 one), nobody could compete with it with those specs at that price and they couldn't for years. And every competitor usually sucked in some major way, most often the trackpad;
2. The Mac Mini is an outstanding piece of hardware for $600. Or was;
3. I've generally found that "Apple tax" complaints levelled against the iPhone to be nothing more than Android cope;
4. The M-series silicon has been an absolute game-changer. I honestly thought the first-generation M1s would be not great but they came out swinging. And the price points for these Macbooks have all been great, much better than the last-gasp-of-Johnny-Ive touch bar butterfly keyboard series, which were objectively awful.
Apple has pretty good competition in every segment with the exception of maybe the iPad, but I'm not a tablet user.
Sure, you can use the App Store and use all the stuff that integrates with iPhone, iCloud, etc
But you can also just treat it as Linux for Laptops (that actually works), and roll with all the standard open source tools.
While they don't _prevent_ Asahi from doing what they're doing, they certainly don't go out of their way to make it easy for them.
But if you don't like it, switch. I don't see vendor lock-in.
Notes sync, Copy/Paste would be hard to give up and took zero effort
And in the rare occasions in which I have to use someone's MacBook, I'm completely lost - like some elderly person.
"Death knell" is a touch hyperbolic. Hardware that can only run quantized models that take up GBs in VRAM falls short of even an A100 (by almost an order of magnitude[0]), which in turn falls short of what an 8xH100 cluster can do (also by another order of magnitude[0]).
I'm an avid believer in local LLMs, but I cannot deceive myself - data center accelerators will win on power dissipation numbers alone[1], even when giving generous allowances for higher efficiency on Apple chips - and assuming the Apple-efficiency advantage persists on the same TSMC process node.
0. Based on my unscientific fine-tuning training experiments across local and rented GPUs. YMMV for inference.
1. Unless Apple surprises everyone and brings back the XServe with M7, if not, then laptop and desktop for factors simply can't dump heat fast enough to compete head-to-head, and will be designed for lower input wattage.
The frontier models are faster, and better at coding, but not so much that i’ll pay $200/month for them.
They use a technique where you only load between 1B and 4B of a 20B dense model for an entire prompt run, not token by token like a MoE, and use mostly the low power ANE instead of GPU cores.
Now, imagine if/when they scale up to 100B or more? On a chip using 2W?
If someone could splinter or fragment the models into more specific tasks i.e "spellchecker AI" and get these working as well as Sonnet 4.6-4.8 on those tasks on a personal laptop. You then question the $100 a month fee.
Bear in mind these laptops are likely to be $5000 or so because of the memory, HDD and M7 chip they likely need.
It feels to me like the beginning of the inflection point but software updates not hardware updates will be the accelerant.
"That’s where EMO comes in.
We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance."
https://allenai.org/blog/emoI want to live in this world too, but these numbers, as of today, are very aspirational and far removed from reality.
I'm no tokenmaxxer; I find my modest local setup useful, I also know the limitations, it's slow and it sucks (relatively) at high-level and/or long-context planning, compared to frontier models. Only a minority of my prompts are max-effort - its not all I do, but, it also means frontier labs aren't dying any time soon
I love local models - I have a machine at home that runs a few for me and it's a lot of fun - but for the time being they are not super trustworthy on tool calls and staying on script. Another year or so might change all that!
*corrected llama version to 3
The weights they “etched” into the FPGA card that’s used for the ChatJimmy demo are that of a Llama 3-something 8b model.
The actually impressive and novel thing is that Taalas’ve managed to automate that process (clearly – nobody transforms 8 billion numbers into a physical representation by hand).
So now, they can work on scaling this process up, and with low enough lead times (I’ll be convinced they have inside connections to TSMC if they can actually deliver on the promised mere 3-4 months delay), will be able to offer 30-100b+ parameter models under half a year after they’re released, at thousands of tokens per second while probably drawing less wattage (per token, not sure about overall).
Exciting times ahead, folks.
The real question is, what are 90% of people going to ask llms to do. I’d argue mostly it’s going to be stuff that works-now or almost-works on local models, but that’s just an opinion. It also depends on the frontier models hitting a wall of steeply diminishing returns, since they set the expectations for all of this stuff - my gut says that’s happened already they just won’t admit it for a while - but we’ll see.
Sometimes, I need a quick throwaway bit of python. That can take 30 minutes of my time.
Apple is the only player here where it would play into their natural hardware incentive to get you to pay more for better hardware. It would make sense for them to find a way to run LLM locally (eg, newer architectures that others here have pointed out).
Interesting times.
Of course, these are a lot of ifs.
There is some signal that this is possible through both hardware innovation and training/data improvements.
Cloud models have their own constraints - I can’t have opus4.8 spend 4 hours on a deep research question I had in the shower without spending money. I can’t do real time video game upscaling and graphics work in the cloud period.
A laptop is about an order of magnitude cheaper than a cloud server thanks to economies of scale, uptime requirements, and other factors.
I'm not talking local Gemma/Qwen vs cloud Opus, but against OpenRouter same Gemma/Qwen
there are reasons to run local - privacy, availability, but cost is not one of them
There has been a lot of market-subsidy in AI which is starting to fade away: e.g. the copilot quotas/pricing. When VC switches from investing to wanting a return, the price equation is likely to change.
You buy a big GPU, you serve LLMs, you print money.
Could you give an example with real figures?
for me it would be about $2 per day in electricity to generate 8 mil tok of Gemma4-26B at 4 bit quantization. this is excluding how much the GPU cost (no amortization)
ignoring the fact that I could get more free tokens per day for this model from Google/OpenRouter, it would cost $4 per day on OpenRouter if paid, but they would run it at full 16 bit precission
this would be the most "profitable" model for me
for Gemma4-31B I can generate only 1 mil tok per day, and so I pay more to get less quality than OpenRouter (ignoring that this model is also free on Google)
Now, this all brings the upfront costs way up, the solar panels are cheap, its all the rest around them that tends to cost money.
Benchmarks maybe? Real world, no.
You just need the context otherwise. There's no way around it.
Whether such a model exists or not is a different question.
So yeah, commercially it might be a death knell. Yes there's still a market for super computers, but would your rather own Apple or Cray?
I would consider an HPE tower server with a processor on the same league as an M6 or M7 under the Cray brand.
At some point there will be diminishing returns towards the "just throw more RAM at it" approach the current frontier models are taking. Commoditization is just as inevitable as it ever was... and in doing so will enable actual leaps of what AI/ML is capable of. That's not to say there won't be a place for 99.999999% accurate vs 99.99999% but those cases will be limited and likely prime to disruption based on real innovation vs access to capital.
SOCs with unified memory have shifted this a bit forward, but they're also expensive as shit.
10TB ram in a consumer device is simply not happening in the next 10 years.
Is this true though? I don't really have time to do the research and have no dog in the race but I'm sceptical...
But then again I'm not there one making our profiting from the claims that there is billions being invested into infra...
Enjoy paying $1000 or more for a little 4 GiB cloud terminal that connects you to all your online accounts where all your actual work gets done. This is the future.
This is highly doubtful.
Rule of thumb: everything people think is exponential is actually an S curve.
What’s frontier now is prosumer in a couple years and commonplace in a couple more.
That's the today hardware.
Now suppose Apple goes to any of Samsung/Micron/Hynix and says "we'll pay you the entire cost of building another DRAM fab and in exchange we want its entire output" and then releases M7 devices with enough memory and compute to run bigger models.
> Unless Apple surprises everyone and brings back the XServe with M7, if not, then laptop and desktop for factors simply can't dump heat fast enough to compete head-to-head, and will be designed for lower input wattage.
Laptops maybe. Desktops can dissipate more heat than the amount of electricity you can draw from a typical household wall outlet.
It's revealing that they aren't doing this: no one wants to fund that gamble on the state of AI demand 12-18 months out, but ate happy to capitalize on their current product lines/capacity.
> Desktops can dissipate more heat than the amount of electricity you can draw from a typical household wall outlet
100% agree, but the data center power and cooling infra are not limited by home wiring, and go way beyond what a wall outlet can safely provide (1,440W max on a typical 15A circuit at 120V). A single H100 maxes out at 700W
They are doing this. All three of the DRAM manufacturers are actively expanding capacity. It's just not an instantaneous process.
> 100% agree, but the data center power and cooling infra are not limited by home wiring, and go way beyond what a wall outlet can safely provide (1,440W max on a typical 15A circuit at 120V). A single H100 maxes out at 700W
A desktop with the performance and the 160GB of VRAM of two H100s would sell like hotcakes. The power consumption is also largely proportional to the amount of compute rather than memory, and can be significantly improved by using lower clock speeds and then either making up for it with wider silicon or just not caring that it can run the same model but is e.g. 25% slower than hardware that uses twice as much electricity.
why not just say "I think that"
do you see yourself as some kind of visionary about this particular topic? literally EVERYONE is saying that, it's the most obvious fact about AI
So maybe they were assholes.
(All the usual caveats about geekbench scores apply, but they're not nothing.)
If people can get opus4.6/gpt5.5-like models locally, labs could raise their prices and sell token speed, better reasoning, mobile-focused improvements, you name it.
Not all consumers are power users and many will be happy to pay for flexibility.
I do wish they have Xserve back, or a Mac Pro that is Rack Based and support multiple node with M6 Ultra. The Hyperscaler market is so large along with AI their old business case of Xserve didn't make sense no long hold true.
That means they're actually incentivized at least short term, to benefit PCs becoming strong enough to do local LLMs. Which makes this play make even more sense. Though, I've been saying for a while that the local AI inflectiom point is the death knell for these frontier labs.