nicoinstrument

Thanks for the suggestion! Have you found that Ray Serve’s built-in autoscaling plays nicely with custom SLO-based concurrency limits, or do you usually let Ray handle the load balancing entirely?"

I'm learning about inference by running vLLM on a k8s cluster (EKS), building a gateway to keep a <2s TTFT SLO.

Most recent ha-ha moment: I kept wondering if it was normal that my cluster was only able to process 4 requests per second per vLLM engine (just seemed really low to me).

I realized a better metric is in-flight requests... Each engine is processing 70 requests at any given time, streaming tokens for over 30s.

Code: https://github.com/Nicolas-Richard/vllm-on-eks

Chime | Software Engineer, Infrastructure | Full Time | REMOTE (US)

We’re looking for talented engineers who: • Have hands-on experience in infrastructure or platform engineering. • Enjoy working on distributed systems, scaling, automation, and reliability. • Bring a passion for operational excellence and continual improvement. • If this sounds like you, check out the position here: https://lnkd.in/gxz2Kac8

Thank you! 1. I think it can be compared to a process of rendering the charts on your local machine to preview the changes and possibly pasting them in your PR for the reviewer. Sort of like mani-diffy by fully manual, yuck. 2. The other obvious comparison is with a process where diffs are reviewed in the ArgoCD web UI, assuming you have all your ArgoCD apps in "autosync: no" mode. This would be after the PR is merged, thus fracturing the review experience, not ideal. It would also cripple ArgoCD a little as the autosync feature is very key to the experience.

you can watch it full length (~1hr) with a San Francisco library account : https://sfpl.bibliocommons.com/v2/record/S93C4825237