Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
by wskwon on Tuesday June 20, 2023 @08:02PM
(#36412503)
Yes, vLLM focuses on maximizing throughput when the VRAM is fully utilized. Nevertheless, I believe users can still benefit from vLLM even if they don't utilize the memory to its full capacity, because vLLM also includes other optimizations orthogonal to the PagedAttention (e.g., optimized CUDA kernels).
[ Parent
| on HN ]
There are no comments on this comment yet.