2. Why would anybody be satisfied with a 64GB model when GPT-4 or 5 or 6 might even be using 1TB of RAM?
3. That may not be the case. With every day that passes, it becomes more and more clear that large LLMs are not that easy to build. Even Google has failed to make something competitive with OpenAI. It's possible that OpenAI is in fact the new Google, that they have been able to establish permanent competitive advantage, and there will no more be free commodity LLMs than there are free commodity search engines.
Don't get me wrong, I would love there to be high quality local LLMs. I have at least two use cases where you can't do them or not really well with the OpenAI API and being able to run LLama locally would fix that problem. But I just don't see that being a common case and at any rate I would need server hardware to do it properly, not Mac laptop.
I'm really not
I had no desire at all until a couple of weeks ago. Even now not so much since it wouldn't be very useful to me
But the current LLM business model where there are a small number of API providers, and anything built using this new tech is forced into a subscription model... I don't see it sustainable, and I think the buzz around llama.cpp is a taste of that
I'm saying imagine a future where it is painless to run a ChatGPT-class LLM on your laptop (sounded crazy a year ago, to me now looks inevitable within few years), then have a look at the kind of things that can be done today with Langchain... then extrapolate
And yes I've played with it. It was/is exciting. I can see use cases for it. However none are achievable because the models are (a) not good enough and (b) too legally risky to use.
(B) llama.cpp supports gpt4all, which states that its working on fixing your concern. This is from their README:
Roadmap Short Term
- Train a GPT4All model based on GPTJ to alleviate llama distribution issues.
The point of llama.cpp is most people don't have a GPU with enough RAM, Apple unified memory ought to solve that
Some people have it working apparently:
before I found the repo above I had a naive attempt to get llama running with mps and it didn't "just work" - bunch of ops not supported etc
It's free for the user up to a point, but it costs OpenAI a lot of money.
Apple is a hardware vendor, so commoditization of the software while finding more market segments is definitely something that'd benefit them.
OTOH, if they let OpenAI become the unrivaled leader of AI that end up being the next Google, they end up losing on a topic they wanted to lead for long time (Apple has invested quite a lot in AI, and the existence of a Neural Engine is Apple CPUs isn't an accident)
if OpenAI isn't able to get couple hundred bucks over the typical lifetime of a computer it means the added value they provide is very low (like several times less than Spotify or Netflix for instance), meaning they'll never be “the next Google”.
And if they are it means it make sense to buy it once instead of paying several times the price through subscription.
> The OpenAI APIs are super cheap for a single user needs. I expect them to be at least close to breaking even with their APIs pricing.
“Close to breaking even” means the price you pay is VC-subsidized, the expected gross margin for such kind of tech company is more than 50%. Expect to pay a lot more if/when the market is captive. And this will scale linearly with your use of the technology.
> energy and opportunity costs
What opportunity cost?
Yes, this is a possibility but cloud computing became a commodity.
But I see why people would pay to have their own private and unfiltered models/embeddings.
> if OpenAI isn't able to get couple hundred bucks over the typical lifetime of a computer it means the added value they provide is very low (like several times less than Spotify or Netflix for instance), meaning they'll never be “the next Google”.
They don't have to worry about this today.
> What opportunity cost?
You could utilize the money and the time spent to do other things.
1. the idea would be that now there is a reason to buy loads more RAM, whereas currently the market for 64GB is pretty niche
2. 64GB is a big laptop today, in a few years time that will be small. And LLaMA 65B int4 quantized should fit comfortably
4. LLMs will be a commodity. There will be a free one
6. LLMs seem to avoid the need for finetuning by virtue of their size - what we see now with the largest models is you just do prompt engineering. Making use of personal data is a case of Langchain + vectorstores (or however the future of that approach pans out)