vLLM: 24x faster LLM serving than HuggingFace Transformers

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by george_123 on Tuesday June 20, 2023 @05:58PM (#36411268)

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by zhisbug on Tuesday June 20, 2023 @06:03PM (#36411330)

yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by ynniv on Tuesday June 20, 2023 @06:41PM (#36411735)

If this were submitted as a new comment it would be at the top of the page.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by ipsum2 on Tuesday June 20, 2023 @07:29PM (#36412214)

probably mean agnostic, agonistic implies the opposite.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by zhisbug on Tuesday June 20, 2023 @07:47PM (#36412377)

oops typo

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by baobabKoodaa on Tuesday June 20, 2023 @07:54PM (#36412432)

You mean like, theoretically, in the future? Or you mean today?

[ Parent | on HN ]