This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you keep them in sync the distinction comes down to hardware and lower-end systematic issues.

But most orgs can't realistically have two distinct software systems. How do you create proper isolation or failure mechanisms between them?

I'm guessing this sort of thing is what you mean by their experience with ETOPS.

* * *

It is but it doesn't matter if it is the input message that crashes it. You just need 3 bad message (sooo a message and 2 retries) to crash it whole

or 1 message that crashes all three on the first try vecause you are voting.

In the end you get never around getting some parts of a system bulletproof and reliable. The question is just how big those parts are.

Ideally three models developed to the same spec independently, with majority voting

you could go to a model that verifies the integrity of the data coming in and makes sure the limits on the data are sane before committing it to the db. using a language with strong safety principles (that are not very "hip") like ada or fortran. or you design it so that the system is robust to failure and expects failure like something from teleco like erlang. redundant hardware is fine and great but having them do verification on the data and monitoring the limits of the system is pretty important too.

It’s a fundamental limitation of identical redundant systems that they shave vulnerability to some of the same threats, particularly bad inputs and capacity issues. It’s important to understand it’s only giving you physical redundancy, such as if one data centre goes down. But the same software bugs, the same bad input data, even the same memory overruns are likely to hit both systems. It’s not bad design, it’s just you have to understand what resiliency you have and plan against each of various such threats according to your risk appetite.

It's very much like the advice that mirror RAID is not a backup solution. That seems to be almost literally what went wrong here.

> they wrote ETOPS regulations ... then failed to apply that to their software

you are pointing out that they prioritized that planes in the air could land safely over allowing more planes to take off. I'm actually quite reassured now.