I hate to pooh-pooh it for everyone, but this was possible before LLaMa. GPT-J-125m/6b have been around for a while, and are frankly easier to install and get results out of. The smaller pruned model even fits on an iPhone.
The problem is more that these smaller models won't ever compete with GPT-scale APIs. Tomorrow's local LLaMa might beat yesterday's ChatGPT, but I think those optimistic for the democratization of chatbot intelligence are setting their hopes a bit high. LLaMa really isn't breaking new ground.
There are all kinds of things I want to be able to do with a LLM that are a lot tighter than general chatbots.
I'd love to see a demo of GPT-J on an iPhone!
The best models will be by API but there is a threshold of utility for LLMs, below which they're just bad story telling toys, that hasn't been available for home users until recently, with FlanT5s and LLaMas.
I’m sure someone did the work you suggest to let folks run on this on more reasonable setups- but I never saw it on the front page here.
Did they achieve GPT-3 performance on anything at that model size? Happy to be corrected, but my understanding was llama does in fact outperform GPT-3 even at the smaller parameter-counts on certain downstream tasks due to using the chinchilla training regime (and other tricks).
I'm not too worried about CLI shenanigans, because of what happened with whisper.cpp - it resulted in apps like https://goodsnooze.gumroad.com/l/macwhisper - wouldn't be at all surprised to see the same happen with llama.cpp
A regular M1 with 8GB of RAM appears to be good enough to run that 7B model. I wonder at what point it will run on an iPhone... the Stable Diffusion model was 4GB when they first released it, and that runs on iOS now after some more optimization tricks.
For me though, the "Stable Diffusion" moment isn't necessarily about the LLaMA model itself. It's not licensed for commercial use, so it won't see nearly the same level of things built on top of it.
The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I know it can be done! Now I just need to wait for the inevitable openly-licensed, instruction-tuned model that runs on the same hardware.
It's that, but also the forthcoming explosion of developer innovation that a local model will unleash. llama.cpp is just the first hint of that.