vLLM: 24x faster LLM serving than HuggingFace Transformers

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by SimFG on Tuesday June 20, 2023 @11:09PM (#36413909)

Cool, I prefer the OpenAI-Compatible api. Although this is not very technically difficult, it is really intimate, because it make me feel free to use all ChatGPT applications.

[ Parent | on HN ]

Re:vLLM: 24x faster LLM serving than HuggingFace Transformers by wskwon on Tuesday June 20, 2023 @11:54PM (#36414206)

Thanks! Please try it out and share any feedback you might have.

[ Parent | on HN ]