Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right?

Depends on how many nines of uptime you need, of course, and other such things.

(Obviously, there are reams of cases where this isn't sufficient)

In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes.

Thankfully I had both onsite and offsite backups; but it took a couple of weeks to order the new equipment, install it and set everything up again.

Now I "devops" all the things and always have the ability to spin up all the services and restore from backups within minutes. This is the new standard in my opinion.

Two questions:

1. How would power conditioners and lightning arresters handle these sorts of situations?

2. Was this using VMWare fault tolerant VMs, just out of curiosity? (It continually live-migrates one VM to a hot spare over a 10G link so either can disappear and the VM continues to run.) Or was this a bespoke application architecture implementing effectively the same thing?

I think it's always better to have the mirror be located in a separate datacenter and then connect both via a SDN.

Why not both?

Yeah, there definitely needs to be an offsite spare for sure.

Why isn't "we're going to risk downtime to speed up our BTD loop for cheap" a good answer?

I've made that very decision.

Partly forced due to internal resource constraints. However, swapping getting a working system out and tested with real users instead of waiting an undefined time to get high availability didn't lose me any sleep.

It's also the case that some systems can withstand a degree of downtime that others can't, or it's not worth paying the cost for the perceived benefit gained.