Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on How Is LLaMa.cpp Possible?. Parent comment
That seems simplistic, but its really simple as that. Naive 3 bit quantization will turn llama 7B into blubbering nonsense.
But llama.cpp quantization is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation:
https://github.com/ggerganov/llama.cpp/pull/1684
Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B quant, for instance, is about as dumb as 7B F16, but the dropoff is not nearly as severe from 3-6 bits.
There are no comments on this comment yet.
That seems simplistic, but its really simple as that. Naive 3 bit quantization will turn llama 7B into blubbering nonsense.
But llama.cpp quantization is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation:
https://github.com/ggerganov/llama.cpp/pull/1684
Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B quant, for instance, is about as dumb as 7B F16, but the dropoff is not nearly as severe from 3-6 bits.