wskwon
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @11:54PM (#36414206)
Thanks! Please try it out and share any feedback you might have.
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @08:13PM (#36412592)
Thanks for the explanation! I believe the two ideas are basically orthogonal. FlashAttention reduces memory read/writes, while PagedAttention reduces memory waste.
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @08:02PM (#36412503)
Yes, vLLM focuses on maximizing throughput when the VRAM is fully utilized. Nevertheless, I believe users can still benefit from vLLM even if they don't utilize the memory to its full capacity, because vLLM also includes other optimizations orthogonal to the PagedAttention (e.g., optimized CUDA kernels).
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @06:41PM (#36411734)
We used matplotlib for the performance charts, and used a free website to convert the slides to the animation gifs.
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @05:56PM (#36411242)
Not really. vLLM optimizes the throughput of your LLM, but does not reduce the minimum required amount of resource to run your model.
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @05:37PM (#36411028)
You can think of LMSYS Vicuna: https://chat.lmsys.org as our hosted demo, as it actually uses vLLM as the backend.
vLLM: 24x faster LLM serving than HuggingFace Transformers
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @03:17PM (#36409083)
vLLM has been adopted by LMSYS for serving Vicuna and Chatbot Arena.