Re:Universal LLM Deployment Engine with ML Compilation
by ruihangl on Friday June 07, 2024 @03:06PM
(#40611833)
A unified efficient open-source LLM deployment engine for both cloud server and local use cases.
It comes with full OpenAI-compatible API that runs directly with Python, iOS, Android, browsers. Supporting deploying latest large language models such as Qwen2, Phi3, and more.
It comes with full OpenAI-compatible API that runs directly with Python, iOS, Android, browsers. Supporting deploying latest large language models such as Qwen2, Phi3, and more.