News for hackers. Stuff that matters.
Re:Using LLaMA with M1 Mac and Python 3.11 by cypress66 on Sunday March 12, 2023 @11:01PM (#35130417)
The performance loss is because this is RTN quantization I believe. If you use the "4chan version" that is 4bit GPTQ, the performance loss from quantization should be very small.
Re:Using LLaMA with M1 Mac and Python 3.11 by xdennis on Sunday March 12, 2023 @11:39PM (#35130694)
What's the 4chan version?
Re:Using LLaMA with M1 Mac and Python 3.11 by aseipp on Monday March 13, 2023 @12:00AM (#35130860)
See https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)
Re:Using LLaMA with M1 Mac and Python 3.11 by cypress66 on Monday March 13, 2023 @02:38AM (#35131944)