(I created an account just to upvote and comment on this)

Absolutely, nevermind managing OS upgrades and needing to use configuration management to mitigate against drift. It took a lot of effort to make sure that each server was not a special snowflake that could not be reliably reproduced.

Also, dealing with vendor warranties, and being on hold with HP (or whoever), then assuring them you're running the latest firmware.. please for the love of god just replace the failed memory/disk/cpu.

I found the sheer physicality of computing infrastructure to be a source of exhaustion and burnout. It's a big part of the reason why I'm a software developer now :)

> Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two.

This is not always the case. For computationally-focused workloads like the OP describes, without direct customer interaction, it may be reasonable to accept risk of downtime in the event of failure. If you are doing computations that take weeks to complete, and you checkpoint regularly, does it really matter if your computation finishes on Saturday morning or Monday morning? If not, you can probably accept 6 hours of downtime once every couple of years and eliminate all of the redundancy overhead described above.

In HPC, the general rule of thumb is buy your hardware if you can be sure you'll run compute on it more than ~40% of the time.

In the bad old days we bought our own servers and rented rack space in a datacenter they took care of all the hardware management, we looked after the software.

Mind you this was only about 4U worth.

For on prem. Or you put it in a colo, and let someone else deal with those things, and you still pretty much the same cost savings vs. cloud - sometimes more (e.g. if you can pick a colo somewhere with cheaper electricity you can save; if you can pick one somewhere with cheaper land, you can save). Or you move to managed hosting and still bare metal, and stop worrying about the physical servers at all, at loss of some freedom, and still get around 90% of the cost savings vs. cloud.

I had a cabinet at the colo data center that was formerly Enron's data center - super thick pipe to the public Internet, in a data center built to survive nuclear war. I had 17 servers and a Federal Reserve quality hardware firewall, total cost of hardware was about $55K, the colo expense of $600 a month, and my team of two spent perhaps a day a month on maintenance. This was in contrast to $96K per month for the same setup at Amazon.