despite best efforts, sometimes the generators don't kick in. sometimes not all of them kick in. sometimes one ATS in one feed in one section that just happens to house one half of an important switch chassis doesn't work as it did during tests last quarter.

True. But Data centers usually have an A and B power systems independent of each other for redundancy . Equipment is usually connected to both. The failure of either should not cause an outage. (Unless some one is stupid enough to connect core network equipment to only A or B power)

Sometimes only A power or B power is delivered to the rack (otherwise you could receive different phases).

Power failures also occur sometimes when the issue is on the wrong side of where power is distributed - we've lost power to our servers when maintenance has gone wrong for example.

From my point of view if you end up with mission critical equipment connected only to A or B power then its engineered wrong.

Very true, but sometimes this happens at fault of the facility, not the affected company. I've been in a facility experiencing widespread power issues because customers were incorrectly delivered A+A or B+B power, instead of A+B.

This is why I'm a proponent battery floated of DC Power for mission critical systems.

Exactly. You can only test for these scenarios to a certain extent and even when everything works as expected during tests, things still tend to break during production when shit hits the fan.

Like you, I'm surprised.

I thought major data centres had redundant power from the grid too (not just battery and generator), so the UPS smooths the power transition as the switching gear fails over to the alternate grid connection; then generators come up if that fails too? Is that considered over-engineering?

They usually do, but these only protect from grid failures. HEX certainly does have battery + generators (it's one of the most important DC's in the country).

Components inside the DC should be redundant, but the customer can decide whether to have dual power feeds from redundant areas of the DC provided to their rack/cage.

Even with all this, issues do happen, Telecity had a very high profile and prolonged issue at the end of last year that affected AWS & Azure amongst others: http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_...

Perhaps a more serious point is that a single power failure (in one data center) can have such a wide impact.