llm-d, Kubernetes native distributed inference

Re:llm-d, Kubernetes native distributed inference by dzr0001 on Tuesday May 20, 2025 @11:49AM (#44042958)

I did a quick scan of the repo and didn't see any reference to Ray. Would this indicate that llm-d lacks support for pipeline parallelism?

[ Parent | on HN ]

Re:llm-d, Kubernetes native distributed inference by smarterclayton on Tuesday May 20, 2025 @12:03PM (#44043121)

We inherit any multi-host support from vLLM, so https://docs.vllm.ai/en/latest/serving/distributed_serving.h... would be the expected path.

We plan to publish examples of multi-host inference that leverages LeaderWorkerSets - https://github.com/kubernetes-sigs/lws - which helps run ranked serving workloads across hosts. LeaderWorkerSet is how Google supports both TPU and GPU multi-host deployments - see https://github.com/kubernetes-sigs/lws/blob/main/config/samp... for an example.

Edit: Here is an example Kubernetes configuration running DeepSeek-R1 on vLLM multi-host using LeaderWorkerSet https://github.com/kubernetes-sigs/wg-serving/blob/main/serv.... This work would be integrated into llm-d.

[ Parent | on HN ]

Re:llm-d, Kubernetes native distributed inference by qntty on Tuesday May 20, 2025 @12:00PM (#44043088)

I believe this is a question you should ask about vLLM, not llm-d. It looks like vLLM does support pipeline parallelism via Ray: https://docs.vllm.ai/en/latest/serving/distributed_serving.h...

This project appears to make use of both vLLM and Inference Gateway (an official Kubernetes extension to the Gateway resource). The contributions of llm-d itself seems to mostly be a scheduling algorithm for load balancing across vLLM instances.

[ Parent | on HN ]