Re:Using LLaMA with M1 Mac and Python 3.11
by vishal0123 on Sunday March 12, 2023 @06:26PM
(#35127100)
Based on my limited runs, I think 4 bit quantization is detrimental to the output quality:
[ Parent
| on HN ]
Yes, I have found 65B quantized to be more nonsensical than 13B unquantized.
The OP article eventually gets around to demonstrating the model and it is similarly bad, zooming from George Washington to the purported physical fitness of Donald Trump?
Re:Using LLaMA with M1 Mac and Python 3.11
by cypress66 on Sunday March 12, 2023 @11:01PM
(#35130417)
The performance loss is because this is RTN quantization I believe. If you use the "4chan version" that is 4bit GPTQ, the performance loss from quantization should be very small.
What's the 4chan version?
See https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)