At Boeing there's a required "failure analysis" document listing all the failure modes and why they won't cause a crash by themselves.

Agreed - This is essentially the corner stone of systems failure analysis - something I wish architects thought about more in the software space.

I'm a product manager for an old (and if I'm being honest somewhat crusty) system of software, the software is buggy, all of it is, but its also self healing and resilient, so while yes, it fails with somewhat alarming regularity, with very lots and lots concerning looking error messages in the logs, but it never causes an outage because it self heals.

Good systems design isn't making bug free software or a bug free system, but rather a system where a total outage requires N+1 (maybe even N+N) things to fail before the end user notices. Failures should be driven by at most, edge cases - basically where the system is being operated outside of its design parameters, and those parameters need to reflect the real world and be known by most stakeholders in the system.

My gripe with software engineers sometimes, they're often too divorced from real users and real use cases, and too devoted to the written spec over what their users actually need to do with the software - I've seen some very elegant (and on paper, well designed) systems fall apart because if simple things like intermittent packet jitter, or latency swings (say between 10ms and 70ms) - these are real world conditions, often encountered by real world systems, but these spec driven systems fall apart once confronted with reality.

> My gripe with software engineers sometimes, they're often too divorced from real users and real use cases

I agree. In my current job, no developer ever talks to users. Projects involve multiple layers of subcontractors, and cross-vendor APIs are part of the bid/spec. Each layer’s purchasing department is in perpetual Bidding Mode while dev teams are already in the middle of doing the actual work. So by default, everyone is Strongly Discouraged From Talking To $VENDOR | $SUBCONTRACTOR | $USER.

16000 - 25000 rads right. Not safe under any circumstance?

Completely safe as long as the block of metal was in place. So you couldn’t just prevent the machine from putting out that much energy, you had to prevent it from doing that without the block in place.

So there should have been an interlocking system

The earlier model that the 25 replaced was all mechanically interlocked. The belief was that software provided that same level of assurance. They performed manual testing but what they weren't able to do was reach a level of speed and fluency with the system to result in the failure modes which caused the issues. Lower hardware costs equals higher profit...