Re:(Databricks cofounder) How Is ChatGPT’s Behavior Changing...
by gyrovagueGeist on Wednesday July 19, 2023 @01:39AM
(#36782486)
If you compiled with llama.cpp with Metal I wouldn’t say its ~all CPU based. But it is still incredible! I need to set this up myself this weekend :)
Note that I never said "open source" just "open models". As in, I can now actually build things with GPT3 capability that run locally. I couldn't care less about the model code.
>Llama 2 still requires multiple kilowatts of electricity and $10,000+ of compute hardware per inference session.
I'm running llama-2-7b-chat on my 8GB M1 Mac right now with llama.cpp. Completions are instant, and essentially at GPT3 levels of accuracy.
The higher param models require up to 64GB RAM, but it's all CPU based.