another_

Check out Determined (https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc.

Disclaimer: I work for Determined.

And general relatively is just the straggler effect because the computation takes longer where mass is concentrated. Empirical evidence the universe is just a large hadoop install.