Oh, I didn’t mean to suggest that the powering of the system was unreasonable. For example, Hetzner will charge you about $250 per rack per month and between 15 and 30 cents (USD roughly) per kWh [1]. I assume they include the cooling and so on, but that means you’re basically looking at a ~4 kW draw for the usual 8xA100 system with some cores and RAM. So budget just under $1/hr for the power and $1000/month all in (depending on location). Most colo power in the US is closer to the Finland calculation than the Germany one.

I’d say keeping a spare system in case of failure is probably a bigger deal than the $10k/yr to house it :).

[1] https://www.hetzner.com/colocation

Eh, the Titan V I bought last year broke even vs AWS inside of a month ($2000 vs $2234).

Obviously there are factors going in to that -- I could live without paying the Tesla tax (didn't need to virtualize, didn't need the vram, did need the fp64), I bought used, I didn't have a problem keeping it fed, I didn't need to burst, etc, but my point is that for some GPU workloads the cloud GPUs are really expensive and the break-even utilization is far south of 100%, more like 5%.

This comes up in nearly every thread, but NVIDIA’s EULA doesn’t permit using their drivers on “consumer” parts in “datacenters”. So you didn’t include the main reason for the Tesla tax: compliance :).

The “consumer” parts are certainly popular ML workstations, and rightly so.

Yep :)

I'm currently renting LambdaLabs V100 instances at 100% utilization for training https://vo.codes

It's really expensive, and I think I should lean into buying hardware at this point.

I want to build a high end GPU rig, but was wondering how easy the setup was. I've only built "consumer" systems before (2x 1080Ti). Is there any appreciable difference?

Do you have a single card? Multiple? What motherboard do you use?

Do you have any takeaways or resources you can share?

I only have one card. I swapped the Titan V with my 1x1080 in my old, cheap motherboard and it just worked. I had to hook up the water cooling, but I did that because it came with a block, not because I optimized the thermal design. To verify motherboard compatibility, I looked up in the nvidia specs how many PCIe lanes and at what speed I should expect and confirmed in HWinfo that they were active in that configuration -- much like I'd look at a network interface to make sure my 10/5/2.5GbE hadn't turned into 1GbE on account of gremlins in the wires.

I'm not using this for machine learning, so you might want to talk to someone who is before pulling the trigger. In particular, my need for fp64 made the choice of Titan V completely trivial, whereas in ML you might have to engage brain cells to pick a card or make a wait/buy determination.

Probably depends on your budget and if you care about support. Building it yourself isn't a big deal at all, but pretty much every major compute vendor will have a GPU platform as well. That's ignoring nvidia themselves who are the easy button and probably your best bet if you think you'll need support up the software stack:

https://www.nvidia.com/en-us/data-center/dgx-station/

That has more to do with Nvidia policy than cloud but I completely agree. For someone who can just plop 4 Geforce GPUs into their PC it's a much better deal than AWS.

The GeForce makes huge difference here.