Re:QLoRA: Efficient Finetuning of Quantized LLMs
by brucethemoose2 on Wednesday May 24, 2023 @09:34PM
(#36066155)
Thats just one method...
- bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now.
- llama.cpp (and derivatives) quantize ahead of time, but its not resource intense.
- mlc llm (vulkan/metal llm inference via tvm) do require lots of ram for their quantization
- bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now. - llama.cpp (and derivatives) quantize ahead of time, but its not resource intense. - mlc llm (vulkan/metal llm inference via tvm) do require lots of ram for their quantization