1. the idea would be that now there is a reason to buy loads more RAM, whereas currently the market for 64GB is pretty niche
2. 64GB is a big laptop today, in a few years time that will be small. And LLaMA 65B int4 quantized should fit comfortably
4. LLMs will be a commodity. There will be a free one
6. LLMs seem to avoid the need for finetuning by virtue of their size - what we see now with the largest models is you just do prompt engineering. Making use of personal data is a case of Langchain + vectorstores (or however the future of that approach pans out)
2. Why would anybody be satisfied with a 64GB model when GPT-4 or 5 or 6 might even be using 1TB of RAM?
3. That may not be the case. With every day that passes, it becomes more and more clear that large LLMs are not that easy to build. Even Google has failed to make something competitive with OpenAI. It's possible that OpenAI is in fact the new Google, that they have been able to establish permanent competitive advantage, and there will no more be free commodity LLMs than there are free commodity search engines.
Don't get me wrong, I would love there to be high quality local LLMs. I have at least two use cases where you can't do them or not really well with the OpenAI API and being able to run LLama locally would fix that problem. But I just don't see that being a common case and at any rate I would need server hardware to do it properly, not Mac laptop.
I'm really not
I had no desire at all until a couple of weeks ago. Even now not so much since it wouldn't be very useful to me
But the current LLM business model where there are a small number of API providers, and anything built using this new tech is forced into a subscription model... I don't see it sustainable, and I think the buzz around llama.cpp is a taste of that
I'm saying imagine a future where it is painless to run a ChatGPT-class LLM on your laptop (sounded crazy a year ago, to me now looks inevitable within few years), then have a look at the kind of things that can be done today with Langchain... then extrapolate
And yes I've played with it. It was/is exciting. I can see use cases for it. However none are achievable because the models are (a) not good enough and (b) too legally risky to use.
(B) llama.cpp supports gpt4all, which states that its working on fixing your concern. This is from their README:
Roadmap Short Term
- Train a GPT4All model based on GPTJ to alleviate llama distribution issues.
The point of llama.cpp is most people don't have a GPU with enough RAM, Apple unified memory ought to solve that
Some people have it working apparently:
before I found the repo above I had a naive attempt to get llama running with mps and it didn't "just work" - bunch of ops not supported etc
It's free for the user up to a point, but it costs OpenAI a lot of money.
Apple is a hardware vendor, so commoditization of the software while finding more market segments is definitely something that'd benefit them.
OTOH, if they let OpenAI become the unrivaled leader of AI that end up being the next Google, they end up losing on a topic they wanted to lead for long time (Apple has invested quite a lot in AI, and the existence of a Neural Engine is Apple CPUs isn't an accident)
if OpenAI isn't able to get couple hundred bucks over the typical lifetime of a computer it means the added value they provide is very low (like several times less than Spotify or Netflix for instance), meaning they'll never be “the next Google”.
And if they are it means it make sense to buy it once instead of paying several times the price through subscription.
> The OpenAI APIs are super cheap for a single user needs. I expect them to be at least close to breaking even with their APIs pricing.
“Close to breaking even” means the price you pay is VC-subsidized, the expected gross margin for such kind of tech company is more than 50%. Expect to pay a lot more if/when the market is captive. And this will scale linearly with your use of the technology.
> energy and opportunity costs
What opportunity cost?
Yes, this is a possibility but cloud computing became a commodity.
But I see why people would pay to have their own private and unfiltered models/embeddings.
> if OpenAI isn't able to get couple hundred bucks over the typical lifetime of a computer it means the added value they provide is very low (like several times less than Spotify or Netflix for instance), meaning they'll never be “the next Google”.
They don't have to worry about this today.
> What opportunity cost?
You could utilize the money and the time spent to do other things.
Doesn't the iPhone use the local processor for stuff like the automatic image segmentation they currently do? (Hold on any person in a recent photo you have take and iOS will segment it)
I think the most glaring situation where this is true is simply one of trust and privacy.
Cloud solutions involve trusting 3rd parties with data. Sometimes that fine, sometimes it's really not.
Personally - LLMs start to feel more like they're sitting in the confidant/peer space in many ways. I behave differently when I know I'm hitting a remote resource for LLMs in the same way that I behave differently when I know I'm on camera in person: Less genuinely.
And beyond merely trusting that a company won't abuse or leak my data, there are other trust issues as well. If I use an LLM as a digital assistant - I need to know that it's looking out for me (or at least acting neutrally) and not being influenced by a 3rd party to give me responses that are weighted to benefit that 3rd party.
I don't think it'll be too long before we see someone try to create an LLM that has advertising baked into it, and we have very little insight into how weights are generated and used. If I'm hitting a remote resource - the model I'm actually running can change out from underneath me at any time, jarring at best and utterly unacceptable at worst.
From my end - I'd rather pay and run it locally, even if it's slower or more expensive.
LLMs will require more than privacy to move locally. Latency, flexibility and cost seem more likely drivers.
I care more about the trust I have to place in the response from the model.
Hell - since you mentioned search... Just look at the backlash right now happening to google. They've sold out search (a while back, really) and people hate it. Ads used to be clearly delimited from search results, and the top results used to be organic instead of paid promos. At some point, that stopped being true.
At least with google search I could still tell that it was showing me ads. You won't have any fucking clue that OpenAI has entered into a partnering agreement with "company [whatever]" and has retrained the model that users on plans x/y/z interact with to make it more likely to push them towards their new partner [whatever]'s products when prompted with certain relevant contexts.
Only people in HN-like communities care about this stuff. Most people find the SEO spam in their results more annoying.
> At least with google search I could still tell that it was showing me ads. You won't have any fucking clue that OpenAI has entered into a partnering agreement with "company [whatever]" and has retrained the model that users on plans x/y/z interact with to make it more likely to push them towards their new partner [whatever]'s products when prompted with certain relevant contexts.
You won't know this for any local models either.
But you will know the model hasn't changed, and you can always continue using the version you currently have.
> Most people find the SEO spam in their results more annoying.
This is the same problem. These models will degrade from research quality to mass market quality as there's incentive to change what results they surface. Whether that's intentional (paid ads) versus adversarial (SEO) doesn't matter all that much - In either case the goals will become commercial and profit motivated.
People really don't like "commercial and profit motivated" in the spaces that some of these LLMs stepping into. Just like you don't like SEO in your recipe results.
Will you? What happens when an OS update silently changes the model? Again this is one of those things only HN-types really care/rant about. I've never met a non-technical person care about regular updates beyond being slow or breaking an existing workflow. Most technical folks I know don't care either.
> This is the same problem. These models will degrade from research quality to mass market quality as there's incentive to change what results they surface. Whether that's intentional (paid ads) versus adversarial (SEO) doesn't matter all that much - In either case the goals will become commercial and profit motivated.
Not at all. Search providers have an incentive to fight adversarial actors. They don't have any incentive to fight intentional collaboration.
> People really don't like "commercial and profit motivated" in the spaces that some of these LLMs stepping into. Just like you don't like SEO in your recipe results.
I disagree. When a new, local business pops up and pays for search ads, is this "commercial and profit motivated?" How about advertising a new community space opening? I work with a couple businesses like this (not for SEO, just because I like the space they're in and know the staff) and using ads for outreach is a pretty core part of their strategy. There's no neat and clean definition of "commercial and profit motivated" out there.
> Almost everyone is willing to trust 3rd parties with data, including enterprise and government customers.
Is absolutely not true. In it's most basic sense - sure... some data is trusted to some 3rd parties. Usually it's not the data that would be most useful for these models to work with.
We're already getting tons of "don't put our code into chatGPT/Copilot" warnings across tech companies - I can't imagine not getting fired if I throw private financial docs for my company in there, or ask it for summaries of our high level product strategy documents.
Saying that cloud models will win over local models is not the same as saying it will be a free-for-all where workers can just use whatever cloud offering they want. It will take time to enterprisify cloud LLM offerings to satisfy business/government data security needs, but I'm sure it will happen.
Yeah but in the cloud that cost is ammortized among everyone else using the service. If you as a consumer buy a gpu in order to run LLMs for personal use, then the vast majority of the time it will just be sitting there depreciating.
iOS actually does already have an offline speech-to-text api. I suspect the reason Siri requires internet is because some downstream part that translates the text into intents/actions is online.
No, it'll be a commodity
Apple wouldn't care if the weights can be extracted if you have to have a Macbook to get the sweet, futuristic, LLM-enhanced OS experience
Edit - I found an example from h.n. user anentropic, pointing at https://github.com/remixer-dec/llama-mps . "The goal of this fork is to use GPU acceleration on Apple M1/M2 devices.... After the model is loaded, inference for max_gen_len=20 takes about 3 seconds on a 24-core M1 Max vs 12+ minutes on a CPU (running on a single core). "
Google had already done some very convincing demos in the last few years well before ChatGPT and GPT-4 captured the popular imagination. Microsoft’s OpenAI deal I would assume will lead to a “Cortana 2.0” (obviously rebranded, probably “Bing for Windows”, “Windows Copilot” or something similar). Google Assistant has been far ahead of Siri for many years longer than that, and they have extensive experience with LLMs. Apple surely realises the position their platforms are in and the risk of being left behind.
I’m also not sure the barrier on iPhone is as great as you suggest - it’s obviously constrained in terms of what it can support now but if the RAM on the device doubles a few times over the next few years I can see this being less of an issue. Multiple models (like the Alpaca sets) could be used for devices with different RAM/performance profiles and this could be sold as another metric to upgrade (i.e. iPhone 16 runs Siri-2.0-7b while iPhone 17 runs Siri-2.0-30b - “More than 3x smarter than iPhone 16. The smartest iPhone we’ve ever made.” etc).
Apple is ahead of the game for a change getting their chips in line as the software exits alpha and goes mainstream.
This is just allowing PyTorch to make use of the Apple GPU, assuming the models you want to train aren't written with hard-coded CUDA calls (I've seen many that are like that, since for a long time that was the only game in town)
PyTorch can't use the Neural Engine at all currently
AFAIK Neural Engine is only usable for inference, and only via CoreML (coremltools in Python)
For 64gb of ram, you can get an m2 pro, or get 96gb which requires the upgraded cpu on the pro. The studio does 64gb or 128gb. But the 128 requires you to spend 5k.
I can't decide between 64 or 96 on m2 pro, and 128 on the studio. Probably go for 96gb. Also what's the impact of the extra gpu cores on the various options? And there are still some "m1" 64gb pros & studios out there. What's the perf difference for m1 vs m2? This area needs serious perf benchmarking. If anyone wants to work with me, maybe I would try my hand. But I'm not spending 15k just to get 3 pieces of hardware.
List prices:
64gb/2tb m2 12cpu/30gpu 14" pro $3900
96gb/2tb m2 max 12/38 14" pro $4500
128gb/2tb m2 max 28/48 studio $5200
…or is it just that the latter had a way too weak iGPU and not enough RAM for AI purposes, whereas the bigger ARM MACs have more GPU power and enough RAM (more than most affordable discrete graphic cards) so that they are usable for some AI models?
> 64GB probably isn't enough for competitive LLMs anyway
I am trying to charitable, but this is pretty not true. And the hedging in your statement only telegraphs your experience.
2. see above
Should be cheap, or why else are Samsung, Micron and Kioxia whining about losses?
Maybe go for something like Optane memory while doing so.
1. That RAM isn't empty, it's being used by apps and the OS. Fill up 64GB of RAM with an LLM and there's nothing left for anything else.
2. 64GB probably isn't enough for competitive LLMs anyway.
3. Inferencing is extremely energy intensive, but the MacBook / Apple Silicon brand is partly about long battery life.
4. Weights are expensive to produce and valuable IP, but hard to protect on the client unless you do a lot of work with encrypted memory.
5. Even if a high end MacBook can do local inferencing, the iPhone won't and it's the iPhone that matters.
6. You might want to fine tune models based on your personal data and history, but training is different to inference and best done in the cloud overnight (probably?).
7. Apple already has all that stuff worked out for Siri, which is a cloud service, not a local service, even though it'd be easier to run locally than an LLM.
And lots more issues with doing it all locally, fun though that is to play with for developers.
I hope I'm wrong, it'd be cool to have LLMs be fully local, but it's hard to see situations where the local approach beats out the cloud approach. One possibility is simply cost: if your device does it, you pay for the hardware, if a cloud does it, you have to pay for that hardware again via subscription.