Nvidia H100 and A100 GPUs – comparing available capacity at GPU cloud providers

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by sashank_1509 on Wednesday June 14, 2023 @07:40PM (#36334435)

If you don't mind answering, what's your use case. Often I feel you either need to do something large scale like a 100 A100s, or you're better off with 8 3090s to run many experiments in parallel

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by bluedino on Wednesday June 14, 2023 @07:50PM (#36334523)

We have a couple GPU nodes in our cluster, each have 4 80GB cards (or V100 nodess are only 32GB).

We have users that have a workstation with a 3090, but the larger amount of memory is what they are after.

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by fpgaminer on Wednesday June 14, 2023 @09:48PM (#36335441)

My biggest project right now is training a multi-label ViT-L/16 model for a few hundred million samples. Mostly a big experiment, so not something I want to invest serious money into.

I have a 2x3090 rig as my local machine, which has been useful for early experimentation, but I'm at the stage now where my runs are at 200 million samples which would take ages on that rig. 8xA100 can do it in tens of hours, which allows me to iterate faster.

An 8x 3090 or 4090 machine locally would be great, but is a huge hassle to build. Last I looked into it there really wasn't a lot of knowledge available online on how to even do it. I did find an EPYC server motherboard and such that I could theoretically use, but couldn't find a great source for a >3,200 Watt server power supply. Everything in that domain is geared towards either B2LargeCorp, or B2ServerBuilders2SmallBusinesses.

I could of course buy an 8xA100 rig no problem for some ungodly amount of money, but as noted above that's not appropriate for this project.

In both cases, I now have to figure out what to do with 3,200kW of heat output in my office which I'm trying to avoid turning into sauna. Or co-lo it for more money out of pocket.

So renting off the cloud has worked well enough at this scale.

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by pmoriarty on Thursday June 15, 2023 @01:05AM (#36336619)

"Mostly a big experiment, so not something I want to invest serious money into."

So how much did it cost?

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by esquire_900 on Thursday June 15, 2023 @01:27AM (#36336756)

You might be interested in http://nonint.com/. He made a number of quite detailed blogposts on his 2 gpu machines (both 8x 3090's), including racks, power delivery etc

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by Tostino on Thursday June 15, 2023 @01:50AM (#36336883)

Thank you for that. Google utterly failed to bring...any relevant results when I was searching for what others were doing for ML rigs.

I'll stick with 2x 3090s for now, but maybe I'll go for an 8x monster in a few years when the hardware shakes out a bit if there is a good reason to do so.

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by extasia on Thursday June 15, 2023 @03:53AM (#36337584)

What's ViT?

[ Parent | on HN ]

Re:Nvidia H100 and A100 GPUs – comparing available capacity ... by knowriju on Thursday June 15, 2023 @06:19AM (#36338475)

I am guessing, Vision Transformer?

[ Parent | on HN ]