News for hackers. Stuff that matters.
Re:Scaling Kubernetes to 7,500 Nodes by mmq on Monday January 25, 2021 @04:44PM (#25909354)
Probably OP was referring to the MPIOperator, TFOperator, PytorchOperator, ... they are under the Kuberflow org, but can be deployed independently of Kubeflow itself. Several other projects are using those operators to provide similar abstractions you mentioned in your blog post, e.g. Gang scheduling, cross-nodes communication, ...

One difference is that these operators use the Kubernetes service interface for communication, generally exposing a headless service for each replica.

Re:Scaling Kubernetes to 7,500 Nodes by sandGorgon on Tuesday January 26, 2021 @12:24AM (#25912888)
@benchess - yes this is what i meant. Using the operator framework.

But more generally, MPI over ssh on a large kubernetes deployment is not a very common pattern. Any reason you chose that ?

Have you looked at Ray or Torch-Elastic (which seems to be officially supported by AWS, etc as well) https://github.com/pytorch/elastic ?