QLoRA: Efficient Finetuning of Quantized LLMs

Re:QLoRA: Efficient Finetuning of Quantized LLMs by dTal on Thursday May 25, 2023 @04:48PM (#36076014)

Why would it need to be loaded fully into system RAM at once? Couldn't it be quantized on the fly, as it's being read from disk?

[ Parent | on HN ]

Re:QLoRA: Efficient Finetuning of Quantized LLMs by MacsHeadroom on Thursday May 25, 2023 @05:58PM (#36076825)

I'm not sure it could be for this complex quantization scheme. It's not a simple round to nearest quantization. Even if it is possible, it would require someone to develop that functionality. Currently it does load the entire unquantized model into RAM first.

[ Parent | on HN ]

Re:QLoRA: Efficient Finetuning of Quantized LLMs by emikulic on Friday May 26, 2023 @11:40PM (#36091887)

Last time I looked into this, the answer was "because huggingface transformers and torch.load are written to do it this way"

You could absolutely do something streaming, or mmap the weights instead of loading them into system RAM. Just the default interfaces don't.

[ Parent | on HN ]