Re:Llama.cpp 30B runs with only 6GB of RAM now
by terafo on Friday March 31, 2023 @05:16PM
(#35393773)
For this specific implementation here's info from llama.cpp repo:
Perplexity - model options
5.5985 - 13B, q4_0
5.9565 - 7B, f16
6.3001 - 7B, q4_1
6.5949 - 7B, q4_0
6.5995 - 7B, q4_0, --memory_f16
According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2].
Perplexity - model options
5.5985 - 13B, q4_0
5.9565 - 7B, f16
6.3001 - 7B, q4_1
6.5949 - 7B, q4_0
6.5995 - 7B, q4_0, --memory_f16
According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2].
[1] https://github.com/qwopqwop200/GPTQ-for-LLaMa
[2] https://arxiv.org/abs/2210.17323