70B Llama 2 at 35tokens/second on 4090

Re:70B Llama 2 at 35tokens/second on 4090 by lawlessone on Wednesday September 13, 2023 @10:52AM (#37497529)

Can you do tuning/training on quantized models? Or are they inference only?

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by woadwarrior01 on Wednesday September 13, 2023 @11:17AM (#37497912)

This paper[1] attempts to do LoRA fine tuning on quantised LLMs. The LoRa weights themselves are full precision, but they're much smaller. Although, I haven't had a chance to try it yet.

[1]: https://arxiv.org/abs/2306.08162

[ Parent | on HN ]