Re:Using LLaMA with M1 Mac and Python 3.11
by cypress66 on Sunday March 12, 2023 @11:01PM
(#35130417)
The performance loss is because this is RTN quantization I believe. If you use the "4chan version" that is 4bit GPTQ, the performance loss from quantization should be very small.
[ Parent
| on HN ]
What's the 4chan version?
See https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)