News for hackers. Stuff that matters.
Re:Universal LLM Deployment Engine with ML Compilation by neetnestor on Friday June 07, 2024 @04:19PM (#40612561)
Looks cool. I'm looking forward to trying building some interesting apps using the SDKs.
[ Parent | on HN ]
Re:Universal LLM Deployment Engine with ML Compilation by yongwww on Friday June 07, 2024 @04:04PM (#40612409)
The MLCEngine presents an approach to universal LLM deployment, glad to know it works for both local servers and cloud devices with competitive performance. Looking forward to exploring it further!
[ Parent | on HN ]
Re:Universal LLM Deployment Engine with ML Compilation by ruihangl on Friday June 07, 2024 @03:06PM (#40611833)
A unified efficient open-source LLM deployment engine for both cloud server and local use cases.

It comes with full OpenAI-compatible API that runs directly with Python, iOS, Android, browsers. Supporting deploying latest large language models such as Qwen2, Phi3, and more.

[ Parent | on HN ]
Re:Universal LLM Deployment Engine with ML Compilation by CharlieRuan on Friday June 07, 2024 @03:37PM (#40612141)
From first-hand experience, the all-in-one framework really helps reduce engineering effort!
[ Parent | on HN ]
Re:Universal LLM Deployment Engine with ML Compilation by cyx6 on Friday June 07, 2024 @03:43PM (#40612187)
AI ALL IN ONE! Super universal and performant!
[ Parent | on HN ]
Re:Universal LLM Deployment Engine with ML Compilation by crowwork on Friday June 07, 2024 @03:18PM (#40611961)
runs on qwen2 on iphone with 26 tok/sec and a OpenAI style swift API
[ Parent | on HN ]
Re:Universal LLM Deployment Engine with ML Compilation by zhye on Friday June 07, 2024 @06:32PM (#40613644)
Glad to see MLC is becoming more mature :) I can imagine the unified engine could help build agents on multiple devices.

Any ideas on how those edge and cloud models collaborate on compound tasks (e.g. the compound ai systems: https://bair.berkeley.edu/blog/2024/02/18/compound-ai-system...)

[ Parent | on HN ]