"connectivity to one of those datacenters is lost"

Anecdotal, but AWS had more global problems than the triple connected data centers I've used in twenty years.

I suffered through many rough times, data center connection or power problems were not (very seldom) one of those.

Most of the problems came from apps that we didn't build well to scale or that had bugs (most frequent cause of problems and sitedowns).

> With AWS I could just set up a multi-availability-zone RDS deployment for the database and an auto-scaling group for the web tier and be confident that AWS will recover the system from most failures

"Confident"? "Most" failures? Are you merely hopeful that the probability of a bad failure is low, or are you able to test the AWS resiliency techniques you mention and to ensure that they stay working? At what cost?

My experience with RDS instances that had multi region failover, was that the failovers worked every time we needed them for deployments (I don’t think we ever needed them for RDS failures). The cost though was enormous. Our write db represented the lions share of our AWS cost, and doubling it for disaster mitigation increased our costs by something like 50%. It was mostly worth it from a business perspective when we were less sure about AWS uptimes, but I’m not sure I could keep justifying the cost given how little problems we had with RDS over time.

Physical hardware failures are handled by having everything in VMs and storage handled by Ceph. We can lose plenty of physical boxes simultaneously before we run into capacity issues.

Multi-DC failover is handled by announcing our public IP block at both locations with different weights. It’s technically active/active because traffic can come in at the secondary DC but we have a internal site-to-site VPN that is used to direct traffic to the primary. If the primary DC goes down the secondary starts handling the traffic instead of passing it along. All the database masters flip to the secondary and things keep humming along.

If we lose the site-to-site then the secondary stops advertising altogether and all traffic is forced to the primary.

So we can lose the site-to-site (which is dedicated) or one of the DCs at any time.

> How do you do failover if a server fails

Not sure if you do not know anything about typical ESXi and vSphere setups, but if a server fails, all virtual machines are automatically migrated to a healthy server. And of course, your HPE G10s are compute only, all storage is on the fiber channel connected SAN.

You don’t get 17 server licenses for vSphere or a SAN for $55k as outlined above. More like $500k plus another $100k/year in maintenance.

You’re still paying a cloud provider, but in this case it’s VMware

Yep! We have a KVM and Ceph based setup but it's basically the same as you describe.