Yes, the maintenance is cheap. The changes are more costly.

We run a dozen bare metal servers and I see the difference what it takes to spin up a new VM vs. set up a new physical server. There's planning, OS installation (we use preseed images but we weren't able to automate everything), sometimes the redundant network setup doesn't play well with what the switches expect (so you need to call the datacenter).

Still, it works out in favor of the bare metal servers. But I'm looking forward to a bit bigger scale to justify a MaaS tool to avoid this gruntwork.

I completely agree that some types of changes are much more expensive with a bare metal architecture than with cloud.

6 years ago, I worked for a company in the mobile space. This was around the time of the Candy Crush boom, and our traffic and processing/storage needs doubled roughly every six months. Our primary data center was rented space co-located near our urban office. For a while, our sysadmins could simply drive over and rack more servers. We reached a point where our cages were full, and the data center was not willing rent us adjacent space. We were now looking at a very large project to stand up additional capacity elsewhere to augment what we had (with pretty serious implications on the architecture of the whole system beyond the hardware) or move the whole operation to a larger space.

This problem ended up hamstringing the business for many months, as many of our decisions were affected by concern about hitting the scale ceiling. We also devoted significant engineering/sysadmin resources to dealing with this problem instead of building new features to grow the business. If the company had chosen a cloud provider or even VPS, it would have been less critical to try to guess how much capacity we'd need a few years down the road to avoid the physical ceiling we dealt with.

Yes, the cloud premium is also a kind of insurance - you know you'll probably be able to double your capacity anytime you need it.

Openstack ironic and bifrost are pretty useful OSS tools for managing baremetal servers.

Yeah, "CAPEX always increases, OPEX always decreases".

[I'm speaking only from the US perspective]

The only difference is if you own the gear or not. If you do own it, then it is CAPEX, and the gear goes on the balance sheet and you can only depreciate it according to the schedules ( in some cases immediately 100% but most of companies blow through that number really quickly ).

In all other cases it is OPEX.

The rule of thumb is that all OPEX can be used to offset the revenue, which is god sent to most companies that aren't printing gobs of money.

So if you make some money and you are past 100% deduction thresholds, when owning gear beefs up the balance sheets and at best slightly decreases taxes while spending money on OPEX significantly decreases taxes.

I didn't parse that in terms of bookkeeping. I took that to mean that the ongoing operational expenses such as, salary for the ops team, dev team, expertise, etc. may be difficult to estimate, once you figure out how to do something, they can be automated.

Under the 2017 Tax Cuts and Jobs Act, you can take a 100% deduction on new and old equipment in the first year. It expires in 2022, but that's because of the shenanigans to avoid formally violating deficit rules, which use a 10-year window for assessing budget impact. I think there's a good chance that it'll be extended as you can game the budget analysis that way indefinitely.

Does that time include security updates for the OS and installed services? I would assume so, but 8 (16?) hours a month seems lower than I'd expect given the frequency of vulnerability discovery and security patches.

Yes.

Thanks!

Everything in one colo facility with no geographic redundancy?

If you look beyond marketing a single top tier DC has better uptime than AWS AZ. You are way more likely to be bitten by AWS control layer issues than by DC in a good location dropping.

Most use cases there is just no need.

Some colo companies, even the small ones, offer multiple datacenters. You then have to either use public IPs for the inter-service traffic, maintain a VPN or contract it from them.

Your suggested solutions aren't reliable at all. AWS/GCP/Azure backbones work absolutely differently.

What about software updates and reboots?

There is the same issue with a cloud provider unless you run immutable deployments. Then, you need to invest in the tooling to produce the immutable images.