so most early stage startups get tens of thousands of dollars of free AWS credits. https://aws.amazon.com/activate/ 100K if ur part of a university accelerator.

it is far far more efficient (as a proportion of time-to-market) to rent and build on top of services.

Kubernetes is where the wider ecosystem is. I dont like it ...but it is what it is.

So Grid.ai is something like AWS Sagemaker. I wanted to figure out what someone can use on a readymade kubernetes cluster.

During my PhD I burned through ~10^5 of compute hours on 4-8 V100 GPU, 128 Core, 1TB+ HPC nodes. Your 10k AWS credits run out really quickly compared to that. And as a startup you are far better of spending some money on a few beefy machines before renting them.

So this thread is a bit of a segue. Even if u rent/buy your machines...my question was "which framework"

I definitely don't want to dispute your calculations.. However, I'm still puzzled what to use.

I would agree with you if the case was some sort of web service in general, but a GPU server that is only used in-house to run some common pyTorch training jobs is very simple to buy and install and run. Those cloud credits are going to get eaten fast as its a few $/hour, assuming you can get them (those programs are only available if you get your financing from a participating incubator or investor). But yeah if you do get 100k for free from someone you should of course use them first :)

Check out Determined (https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc.

Disclaimer: I work for Determined.