We have users that have a workstation with a 3090, but the larger amount of memory is what they are after.
I have a 2x3090 rig as my local machine, which has been useful for early experimentation, but I'm at the stage now where my runs are at 200 million samples which would take ages on that rig. 8xA100 can do it in tens of hours, which allows me to iterate faster.
An 8x 3090 or 4090 machine locally would be great, but is a huge hassle to build. Last I looked into it there really wasn't a lot of knowledge available online on how to even do it. I did find an EPYC server motherboard and such that I could theoretically use, but couldn't find a great source for a >3,200 Watt server power supply. Everything in that domain is geared towards either B2LargeCorp, or B2ServerBuilders2SmallBusinesses.
I could of course buy an 8xA100 rig no problem for some ungodly amount of money, but as noted above that's not appropriate for this project.
In both cases, I now have to figure out what to do with 3,200kW of heat output in my office which I'm trying to avoid turning into sauna. Or co-lo it for more money out of pocket.
So renting off the cloud has worked well enough at this scale.
So how much did it cost?
I'll stick with 2x 3090s for now, but maybe I'll go for an 8x monster in a few years when the hardware shakes out a bit if there is a good reason to do so.
I can't help but feel it's another cloud/enterprise cash grab. The SXM baseboards are way more expensive than server motherboards and NVIDIA makes them.
SXM baseboards are made by more than Nvidia, Dell, HP, and Supermicro all have their own designs.
(Disclaimer - I work for MS Azure but have no internal knowledge about the costs/designs/capex whatever of these systems)
Some automatically drop those limits at a certain age, others require you talk to them.
Power draw. The 4090 pulls 450 watts, and is destroying connectors. H100 SXM uses 700 watts.
If the PCIe consortium wants NVIDIA to use their standard for top end products, they need to make a better connector.
I was talking about the SXM interface instead of PCIe
Also, that’s good feedback on the CPUs bottlenecking. I’ll let our HPC hardware team know about this.
We are also looking into GPU direct storage to help resolve this:
Somewhat related and hopefully helpful: My experience so far using the H100 PCIes to train ViTs:
Pros: A little over 2x performance compared to A100 40GB. 80GB by default. fp8 support, which I haven't played with but supposedly is another 2x performance win for LLMs. For datacenters and local workstations it's twice as power efficient as the A100s. Supposedly they have better multi-GPU and multi-node bandwidth, but my workload doesn't stress that and I can only rent 1xH100s at the moment.
Cons: I had trouble using them with anything but the most recent nVidia docker containers. The somewhat official PyTorch containers didn't work, nor did brewing my own. Luckily the nVidia containers have worked fine so far. I just don't like that they use nightly PyTorch. In addition to that, because of their increased performance, they really start to push the limits on feeding data fast enough to them with existing system configurations. I was CPU-limited on the LambdaLabs 1xH100 machines because of this.
Overall the pricing has worked out equal for my use-case, but I'm sure fp8 would make them more affordable. If fp8 were available out-of-the-box on PyTorch I'd play with it, but it's only available right now from some nVidia codebase specific to LLMs.
Even at equal pricing, having twice the power per GPU and per node is a big win. That increases experiment iteration across the board.
Side note: If I recall correctly, nVidia is heavily differentiating the H100 products by their interface this go around, which I find quite odd. The SXM version of the cards are supposed to be something like twice as beefy as the PCIE version? Not sure why they're doing that; gonna make comparing rentable instances all the more difficult if you overlook that little detail. A100 had a little bit of this, but the difference was never much in practice except between specifically the A100 40 GB PCIe and the 80 GB SXM, which was something like 10% faster.
Anyone else having fun with the new toy our overlords have allowed us to play with?