Disclosure: I work on Google Cloud.

I think the right way to think about the economics here is either “I would pay $X/hr for this short-lived job” or “I want to compare with buying it” (3-yr committed use discount in our case, RIs / Instance Savings Plan for AWS). Unless you are an ML research lab (Google Brain, FAIR, OpenAI, etc.) or an HPC style site sharing these, you won’t get 100% utilization out of your “I just bought it” purchase. Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year.

One amusing thing that’s not clear about “just buy a DGX” is that many people can’t even rack one of these. At 400 watts per A100, our 16x variant is 6.4 kW of GPUs. That’s before the rest of the system, etc. but there are (sadly) a lot of racks in the world that just can’t handle that.

Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the Bottom line of ownership is far bigger and as you say, you really need to get 100% utilisation to capitalise in that level of outlay. Hence the on-demand cloud option on many levels becomes cost effective, even if it does seem not cheap. Sure it is not cheap, but overall, it can and does work out way cheaper for many use cases.

Makes you wonder what kind of power costs you would incur running one 100% utilisation. Certainly even with best prices, be looking at several thousand a year I would of thought, that's not even factoring in provision. Which would mean 3 phase power for that type of load and then you have to balance out the phases. So many little details that become more an issue when you start getting to datacenter level power usage. Then UPS load/capacity costs/planning, networking. So whilst the costs of these units are high, the other costs that add up, sure do add up fast.

Oh, I didn’t mean to suggest that the powering of the system was unreasonable. For example, Hetzner will charge you about $250 per rack per month and between 15 and 30 cents (USD roughly) per kWh [1]. I assume they include the cooling and so on, but that means you’re basically looking at a ~4 kW draw for the usual 8xA100 system with some cores and RAM. So budget just under $1/hr for the power and $1000/month all in (depending on location). Most colo power in the US is closer to the Finland calculation than the Germany one.

I’d say keeping a spare system in case of failure is probably a bigger deal than the $10k/yr to house it :).

[1] https://www.hetzner.com/colocation

Eh, the Titan V I bought last year broke even vs AWS inside of a month ($2000 vs $2234).

Obviously there are factors going in to that -- I could live without paying the Tesla tax (didn't need to virtualize, didn't need the vram, did need the fp64), I bought used, I didn't have a problem keeping it fed, I didn't need to burst, etc, but my point is that for some GPU workloads the cloud GPUs are really expensive and the break-even utilization is far south of 100%, more like 5%.

This comes up in nearly every thread, but NVIDIA’s EULA doesn’t permit using their drivers on “consumer” parts in “datacenters”. So you didn’t include the main reason for the Tesla tax: compliance :).

The “consumer” parts are certainly popular ML workstations, and rightly so.

Yep :)

I'm currently renting LambdaLabs V100 instances at 100% utilization for training https://vo.codes

It's really expensive, and I think I should lean into buying hardware at this point.

I want to build a high end GPU rig, but was wondering how easy the setup was. I've only built "consumer" systems before (2x 1080Ti). Is there any appreciable difference?

Do you have a single card? Multiple? What motherboard do you use?

Do you have any takeaways or resources you can share?

I only have one card. I swapped the Titan V with my 1x1080 in my old, cheap motherboard and it just worked. I had to hook up the water cooling, but I did that because it came with a block, not because I optimized the thermal design. To verify motherboard compatibility, I looked up in the nvidia specs how many PCIe lanes and at what speed I should expect and confirmed in HWinfo that they were active in that configuration -- much like I'd look at a network interface to make sure my 10/5/2.5GbE hadn't turned into 1GbE on account of gremlins in the wires.

I'm not using this for machine learning, so you might want to talk to someone who is before pulling the trigger. In particular, my need for fp64 made the choice of Titan V completely trivial, whereas in ML you might have to engage brain cells to pick a card or make a wait/buy determination.

Probably depends on your budget and if you care about support. Building it yourself isn't a big deal at all, but pretty much every major compute vendor will have a GPU platform as well. That's ignoring nvidia themselves who are the easy button and probably your best bet if you think you'll need support up the software stack:

https://www.nvidia.com/en-us/data-center/dgx-station/

That has more to do with Nvidia policy than cloud but I completely agree. For someone who can just plop 4 Geforce GPUs into their PC it's a much better deal than AWS.

The GeForce makes huge difference here.

You're definitely not going to be able to shove it into just any datacenter, but there's enough of a demand for it where specialty datacentes are popping up.

We're in Colovore, which has a fantastic power density and is running roughly 1000k DGXs in their datacenter. It really wasn't all that difficult to get up and running. For us it made total sense, but we utilize physical hardware completely and have to scale into the cloud fairly regularly.

Right. I’m not saying you can’t, but I am saying that when someone goes to their IT team and says “Hey, we want to buy a few of these” they’ll get met with a groan :).

Did you move all your other gear into Colovore? (That’s one of the challenges, you often need to be close to the rest of your systems / data)

We have a gigabit pipe to AWS and a local NAS for data caching.

We have a similar situation to yours and have had a very positive experience with Colovore.

We also run at 100% utilization 24/7 and frequently use the cloud for burst before we go out and buy more cards/servers.

Colovore was super easy to get up and running and we will save at a minimum hundreds of thousands of dollars with this setup over exclusively using cloud instances.

Yuuuup. Colovore definitely made it a lot easier, and I sing their praises to anyone who will listen.

> Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year.

Oh, so cloud providers are just giving away free resources? How generous!

Seriously, let's do the math:

If a V100 instance cost $9000 new and you bought it a year ago, you could still sell it today for over $3000. On AWS, an instance on a 1-yr CUD costs more than $1400/month, for a total of over $16000. You don't even need 50% utilization to break even. It doesn't matter how fast the A100 is.

In essence, this is a supercomputer.

Kind of. I think a giant pile of them hooked together with good networking would be :).

But also, you can rent a single slice of one for tens of seconds and then walk away. I personally probably do about 20-ish GPU hours a month while dabbling (and only because I’m lazy and have my GPUs attached while I’m working, even if it’s just debugging the Python bits). For a V100 that’s $50/month, which is in the noise compared to dealing with owning infrastructure (and would be even less if I’d get by with a T4).

FYI: NVIDIA is notorious for bullshit benchmarks. There's no way it'll be 2.5x the speed of V100 on any kind of a realistic workload.

Agree. That is theoretical speed up. Most applications are bandwidth bounded thus cannot get that ratio.

I feel like if you’re maxing one of these clusters for an entire month at a time, it might still be cheaper to just buy one.

Ah, but when you use a large cloud, you're not just paying for the cost of the server. You're paying for access to a large high performance network and storage system that is extremely close to your server. Replicating those properties will cost you far more (well, it depends on your specific needs) than the cloud provider.

The DGX A100 costs a quarter million dollars, so the crossover time might be significantly longer than a month.

So six months? Nine months? And at the end, you have an asset. You can always sell it and replace it with the newer model.

We did the math and decided to buy. For us, the break even point (including everything) was nine months of running 24/7.

That's true for any cloud server.

You forgot there is a lot of IT work after buying the hardware to set it up properly. That’s time spent

That’s a one time cost plus maintenance.

Still comes out cheaper than cloud offerings.

untrue

Intel Xeon Gold 5120 14Core @ £1200 used

renting 1Core @ £1.20/mo.

would take 6 years to fully pay for that single core alone, and that's excluding the 512MB RAM, 10GB SSD and unlimited 400Mb/s bandwidth

Here's a completed ebay auction from April 20th that sold a 5120 for $699: https://www.ebay.com/itm/Intel-Xeon-Gold-5120-SR3GD-14-Core-... I know the pound hasn't been doing well lately, but surely it hasn't been that bad.

we're still talking years, not one month

Where are you getting $1.20/mo on 1 core/512MB ram/10GB ssd and half a GB bandiwdth? Amazon for example was something like $50/mo for bandwidth alone

it's not AWS-grade, just a budget VPS, but has been just as reliable and performant so far. it's out there should you care to seek. the correct currency of £ will help

You are renting an overprovisioned timeslice, not a dedicated core. Even Hetzner's time slice instances are twice the price. Their dedicated CPU instances are almost 10x the price.

possibly, but nothing to suggest that 6 months in. they suit the use cases perfectly

It's definitely a timeslice, and that throws off your comparison wildly. Otherwise you'd be able to rent a 32 core dedicated machine at the same place for $48 a month. Can you?

it mentions "dedicated resources", but it's unclear whether this applies to CPU

it might throw the comparison off if there was variation in performance at different times, but there isn't

cores scale with RAM, you cannot rent 32 with 512MB

They're probably talking about dedicated RAM, although even then I wouldn't trust it not to be overprovisioned up to 2:1 on NVME swap.

Hetzner dedicated cores come with 4gb of memory per vCPU for reference.

1.2 GBP/mo for a dedicated core is wildly off market price. There's no way they'd be able to make a profit, so either they're going out of business tomorrow or it's not dedicated.

hourly benchmarks confirm it's definitely DDR4 RAM, standard SATA SSD with acceptable IOPS, and decent enough network. it's the budget branch of the biggest hosting provider so maybe they can afford to. how long it lasts only time will tell

Any chance you can give the link? Can't really find anything :(

Compare it to gattling guns GAU-8. They shoot at 3900 rpm but only for a few seconds. The reason is that a firing solution may only be correct for a split second. It's all about peak performance because you only shoot when you really need to.

The same thing happens with these large instances. Because these instances are so much bigger that you won't use them for the entire month. You'll use them to get results within hours or days.

What's the perf/$ ratio for these different instances ?

I've also read that the A100 can deliver 4x the performance on some DL workloads.