Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by kcorbitt on Monday February 19, 2024 @05:30PM
(#39435609)
This actually already exists! We did a writeup of the relevant optimizations here: https://openpipe.ai/blog/s-lora
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by azeirah on Monday February 19, 2024 @06:12PM
(#39435965)
I recall a recent discussion about a technique to load the diff in weights between a lora and base model, zip it and transfer it on a per-needs basis.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by xzyaoi on Tuesday February 20, 2024 @08:07AM
(#39440736)
There's also papers for hosting full-parameter fine-tuned models: https://arxiv.org/abs/2312.05215
Disclaimer: I'm one of the authors.
I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone.
The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.