*only in the cloud environment

Throw some 3090s in a rack and you’ll break even in 3 months

Because that's "illegal" so cloud providers can't do it.

Can you describe what you mean by that?

The GeForce driver EULA doesn't allow it to be used in servers or something like that, so clouds all have to use the more expensive professional cards.

Disclaimer: I work at Exafunction

I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand.

But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.

Not to mention AWS has had a GPU cloud offering monopoly because Google Cloud and Microsoft Azure were publicly available until 2019.

GCP still provides NVIDIA K80. I wonder is it still worth to hold.

I think you'd probably always want to go with T4's since they are the same price unless there's just no availability for them.

the HPC crowd are not able to add GPUs, that I know of.. deepLearning group of algorithms do kick butt for lots of kinds of problems+data .. though I will advocate that dl is NOT the only game in town, despite what you often read here

In what context? HPC and certain code bases have been effectively leveraging heterogenous CPU GPU workloads for a variety of applications for quite awhile. I know of some doing so in at least 2009 and know plenty of prior art was already there by that point, it's just a specific time I happen to remember.

ok - the academic study in front of me dated 2020 says "no" but it is non-US researchers, public science. I have no reason to believe one way or the other, but I literally read this today.

reading again - it seems this paper calls HPC with GPUs a slightly different name "GPGPU" and lists the research activity separately.. so I didn't see it as HPC; basically what I wrote is not accurate. got it