70B Llama 2 at 35tokens/second on 4090

Re:70B Llama 2 at 35tokens/second on 4090 by checkyoursudo on Wednesday September 13, 2023 @07:02AM (#37495100)

Quantisation tradeoff is an interesting question. My experience so far is that larger models with lower bit quantisation seem to perform better than smaller models with higher bit quantisation. So, e.g., 30B-4bit appears to perform better than 13B-8bit. But, then, an 8-bit quantisation seems not really distinguishable from the non-quantised model, so what benefit might you gain from not quantising the model.

Of course, speed, accuracy, power consumption, etc are all considerations, sometimes at odds with each other.

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by woadwarrior01 on Wednesday September 13, 2023 @07:41AM (#37495411)

It's diminishing returns in terms of perplexity below 4 bits, and this is true even with newer quantisation methods like AWQ and OmniQuant.

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by lhl on Wednesday September 13, 2023 @09:09AM (#37496267)

I think OmniQuant is notable because it shifts the bend of the curve to 3-bit. While < 3-bit still ramps up, it's notable in that it's usable and doesn't go asymptotic: https://github.com/OpenGVLab/OmniQuant/blob/main/imgs/weight...

What EXL2 seems to bring to the table is that you can target an arbitrary quantize bit-weight (eg, if you're a bit short on VRAM, you don't need to go from 4->3 or 3->2, but can specify say 3.75bwp). You have some control w/ other schemes by setting group size, or with k-quants, but EXL2 is definitely allows you to be finer grained. I haven't gotten a chance to sit down with EXL2 yet, but if no one else does it, it's on my todo-list to be able to do 1:1 perplexity and standard benchmark evals on all the various new quantization methods, just as a matter of curiosity.

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by lawlessone on Wednesday September 13, 2023 @10:52AM (#37497529)

Can you do tuning/training on quantized models? Or are they inference only?

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by woadwarrior01 on Wednesday September 13, 2023 @11:17AM (#37497912)

This paper[1] attempts to do LoRA fine tuning on quantised LLMs. The LoRa weights themselves are full precision, but they're much smaller. Although, I haven't had a chance to try it yet.

[1]: https://arxiv.org/abs/2306.08162

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by antupis on Wednesday September 13, 2023 @08:48AM (#37496042)

Yup, but the thing is that can you get model working how you want and can you run it production. My experience is that gpt4 yes, gpt3.5 yes but needs some work Llama2-70b 8-bit or more yes but needs some work after that it really depends usecase.

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by ban-lan-gen on Wednesday September 13, 2023 @10:27AM (#37497220)

Something like that 4^30 >>> 8^13?

[ Parent | on HN ]