News for hackers. Stuff that matters.

A comment on vLLM: 24x faster LLM serving than HuggingFace Transformers. Parent comment

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @05:37PM (#36411028)
You can think of LMSYS Vicuna: https://chat.lmsys.org as our hosted demo, as it actually uses vLLM as the backend.
[ Parent | on HN ]

There are no comments on this comment yet.