News for hackers. Stuff that matters.

A comment on Mistral CEO confirms 'leak' of new open source AI model nearing GPT4 performance. Parent comment

Re:Mistral CEO confirms 'leak' of new open source AI model n... by semmulder on Wednesday January 31, 2024 @04:50PM (#39209894)
FYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0

Also check out the new prefix caching, I see huge potential for batch processing purposes there!

[ Parent | on HN ]
Re:Mistral CEO confirms 'leak' of new open source AI model n... by brucethemoose2 on Wednesday January 31, 2024 @06:47PM (#39211144)
Missed this, thanks.

Everything is moving so fast!

[ Parent | on HN ]
Re:Mistral CEO confirms 'leak' of new open source AI model n... by tgaddair on Thursday February 01, 2024 @03:53PM (#39221320)
Yes, we (LoRAX devs) saw that (we know the author pretty well). It's a useful addition, though quite a bit simpler than our level of support for multi-LoRA inference. We're planning on doing a more comprehensive comparison soon, now that it's officially out.

I will say that if you want to explore the forefront of this multi-LoRA inference, definitely worth giving LoRAX a look. We just added support for per-request model merging (https://predibase.github.io/lorax/guides/merging_adapters/) as an example, and are planning on continuing to double down on this idea of combining adapters in some pretty unique ways.

[ Parent | on HN ]