Even with that build it might still take several days to train some deep learning models. Much better would be multiple machine distributed model training. You can pay the true cost either by waiting for your one machine model to train, by setting up complicated infrastructure on your own (either in house or in the cloud) or by using a cloud/third party distributed computing option.

> by setting up complicated infrastructure on your own (either in house or in the cloud) or by using a cloud/third party distributed computing option.

The argument is/would be, if your business is heavily based upon deep learning and is that compute heavy on a hourly basis, you may in fact save millions daily by having your own in-house infrastructure.

It seriously isn't complicated to run as much as AWS and co want to say it is to push their own sales. But it does cost capital investment and maybe 1 or 2 IT guys.

There's an inflection point where it makes sense.

What do you expect the salary is for an IT guy that knows how to set up and maintain distributed machine learning infrastructure on prem?

Not that much. It's really no different than setting up other servers. You can also rent services instead of maintaining a FT position.

IT has been commoditized, it's not a special domain of expensive and rare experts anymore.

Don't fool yourself, it is not hard.

For about three times the prices you can get double that performance in one machine (dual socket with 8-10 GPUs). You can get a lot done with that, and in some cases waiting days is actually acceptable. If you need more power than that it does indeed get more complicated and the trade-off might change.