Re:Punica: Serving multiple LoRA finetuned LLM as one
by Palmik on Thursday November 09, 2023 @01:21AM
(#38201792)
Added to my reading list! The world of quantizations is moving so fast even TheBloke might not be able to keep up!
So Atom base models would be compatible with Punica?
I also wonder, many people already train LoRAs in 8 or even 4 bit (for the base model), would it make sense to match the quantization algo used during training and inference?
So Atom base models would be compatible with Punica?
I also wonder, many people already train LoRAs in 8 or even 4 bit (for the base model), would it make sense to match the quantization algo used during training and inference?