vLLM: 24x faster LLM serving than HuggingFace Transformers

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by marcopicentini on Tuesday June 20, 2023 @05:33PM (#36410974)

Is it available an hosted demo?

What are use cases for which open source models are equivalent of GPT 3.5?

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @05:37PM (#36411028)

You can think of LMSYS Vicuna: https://chat.lmsys.org as our hosted demo, as it actually uses vLLM as the backend.

[ Parent | on HN ]