News for hackers. Stuff that matters.

A comment on vLLM: 24x faster LLM serving than HuggingFace Transformers. Parent comment

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @05:56PM (#36411242)
Not really. vLLM optimizes the throughput of your LLM, but does not reduce the minimum required amount of resource to run your model.
[ Parent | on HN ]
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by e12e on Wednesday June 21, 2023 @03:41AM (#36415595)
But (in theory) - llama.cpp could implement similar approach to paging/memory and see a speedup for 4bit models on cpu?
[ Parent | on HN ]