Check out Determined (https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc.

Disclaimer: I work for Determined.