News for hackers. Stuff that matters.

A comment on How Is LLaMa.cpp Possible?. Parent comment

Re:How Is LLaMa.cpp Possible? by YetAnotherNick on Wednesday August 16, 2023 @02:37AM (#37143536)
The effect is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684
[ Parent | on HN ]
Re:How Is LLaMa.cpp Possible? by astrange on Wednesday August 16, 2023 @03:17AM (#37143764)
This paper from last month has a method for acceptable 3-bit quantization and a start at 2-bit.

https://arxiv.org/abs/2307.13304

[ Parent | on HN ]