I found something interesting via Github, Enova open source.

1. Easily deploy LLM and create a WebUI with just a few command lines! 2. Recommend optimal computing resources and operating parameter configurations for LLM 3. Dive deep into LLM's operating status and let it self-heal during any abnormal conditions! 4. Achieve seamless load balancing through autoscaling!