Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by kkielhofner on Tuesday March 14, 2023 @10:30AM (#35152116)

llama.cpp doesn't use the GPU at all. The genius *.cpp (whisper.cpp, llama.cpp) projects are specifically intended to optimize/democratize otherwise GPU only models to run on CPU/non-GPU (CUDA, ROCm). Technically speaking the released models are capable of running on GPU via standard framework (PyTorch, TensorFlow) support for CPU but in practice without a lot of optimization they are incredibly slow to the point of useless, hence *.cpp.

You want something along these lines (warning: unnecessarily potentially offensive):

https://rentry.org/llama-tard-v2

[ Parent | on HN ]

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by aortega on Tuesday March 14, 2023 @11:34AM (#35153191)

Llama.cpp takes advantage that LLaMa 7B is a tiny, very optimized model. It would run in anything, and very fast. I really doubt you can run the 30B or 65B models at acceptable speed on a CPU at least for a couple years. (I'm ready to eat my words in a couple weeks)

[ Parent | on HN ]

Re:Alpaca- An Instruct Tuned Llama 7B. Responses on par with... by zitterbewegung on Tuesday March 14, 2023 @11:54AM (#35153549)

Okay my thread ripper can handle it because it has a 128GB of Ram.

[ Parent | on HN ]