Re:Llama.cpp 30B runs with only 6GB of RAM now
by bakkoting on Friday March 31, 2023 @06:06PM
(#35394332)
I'd guess the GPTQ-for-LLaMa repo is using a larger context size. Poking around it looks like GPTQ-for-llama is specifying 2048 [1] vs the default 512 for llama.cpp [2]. You can just specify a longer size on the CLI for llama.cpp if you are OK with the extra memory.
[1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8e...
[2] https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bd...