vLLM: 24x faster LLM serving than HuggingFace Transformers

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by brucethemoose2 on Tuesday June 20, 2023 @06:46PM (#36411778)

Reading between the lines, it sounds like some of the speedup comes from VRAM savings on an otherwise close to full GPU?

This is definitely cool and needed, but it might not be so dramatic running 3-5 but quant on a less full GPU.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @08:02PM (#36412503)

Yes, vLLM focuses on maximizing throughput when the VRAM is fully utilized. Nevertheless, I believe users can still benefit from vLLM even if they don't utilize the memory to its full capacity, because vLLM also includes other optimizations orthogonal to the PagedAttention (e.g., optimized CUDA kernels).

[ Parent | on HN ]