News for hackers. Stuff that matters.

A comment on vLLM: 24x faster LLM serving than HuggingFace Transformers. Parent comment

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by zhisbug on Tuesday June 20, 2023 @06:03PM (#36411330)
yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.
[ Parent | on HN ]
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by ynniv on Tuesday June 20, 2023 @06:41PM (#36411735)
If this were submitted as a new comment it would be at the top of the page.
[ Parent | on HN ]
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by ipsum2 on Tuesday June 20, 2023 @07:29PM (#36412214)
probably mean agnostic, agonistic implies the opposite.
[ Parent | on HN ]
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by zhisbug on Tuesday June 20, 2023 @07:47PM (#36412377)
oops typo
[ Parent | on HN ]
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by baobabKoodaa on Tuesday June 20, 2023 @07:54PM (#36412432)
You mean like, theoretically, in the future? Or you mean today?
[ Parent | on HN ]