Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by disiplus on Friday December 01, 2023 @05:32AM
(#38485265)
i have lenovo legion with 3070 8GB and was wondering should i use that instead of my macbook m1pro.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by tarruda on Friday December 01, 2023 @06:09AM
(#38485476)
The main focus of llama.cpp has been Apple silicon, so I suspect M1 would be more efficient. The author recently published some benchmarks: https://github.com/ggerganov/llama.cpp/discussions/4167
With my system, Llama.cpp can run Mistral 7B 8-bit quantized by offloading 32 layers to the GPU (35 total) at about 25-30 tokens/second, or 6-bit quantized by offloading all layers to the GPU at ~ 35 tokens/second.
I've tested a few 13B 4-bit models such as Codellama and got about 10 tokens/second by offloading 37 layers to the GPU. Got me about 10-15 tokens/second.