Well the cost of the gpus themselves is insane. A single h100 you can hold on your hand costs the around the same as an entire Tesla model 3.

Less chance of fatal-to-operator malfunctioning tho.

Anyone have any predictions about when the prices will start going down significantly? Anyone think we could see price per hour per GPU cut in half by July?

When some more EUV fabs come online. So... 5-10 years min.

Looks like we have a bunch of start-up cloud companies who have money being thrown at them by god knows who buying thousands of H100s and other GPUs. These are all going to come online around the same time in early spring. I think we will see prices drop rapidly by June .

$30K<->$40K and it apparently costs Nvidia around $3.3K to manufacture an H100 GPU for info

Cost to produce doesn't matter. The reason these things are so expensive is demand and the limited capacity of fabs to produce them.

You pay mostly for CUDA, not the GPU

Yeah, looks like you’re expected to spend a minimum of $3,000,000 to meet the minimum requirement for some of the offerings.

Isn't 2-3$ per hour the standard rate for A100s and H100s spot pricing?

They're not only insane (i.e. offering H100s for rent at a quarter the price they'd be to purchase) they're also impossible, since every listing claims to have deployed an Infiniband fabric speed which is firmly in the future.

Jonathan from TensorDock (https://tensordock.com/) here - we listed two of our A100 and H100 clusters on the site.

The IB equipped on our clusters (can't speak to others) is 8x 400 Gbps. Most customers training foundational models are able to fully utilize that fabric in parallel.

Which HCAs are enabling that? You're using eight 4-link QSFPs here?, presuming this is NDR?

And out of curiosity, is aggregate bandwidth the normal marketing metric in this industry? In my neck of the woods this would be reported as an NDR400 system.