But would you download an alternative distribution made by a group of enthusiastic aeronautics professionals that has been used by tons of other people with no problems? Think Cyanogenmod.
In this plane software example if I was a plane manufacturing org I could dedicate people to integrate it and test it.
There is a difference between an organization dedicated to making a product and a group of hobbyists coming together to scratch their itch. There is a scale, a threshold, above which you need big-org organization and sharing of responsibilities.
Small expert teams are fantastic for the sort of projects that can be done by small expert teams. For larger things there needs to be a bit more infrastructure and organization, or at least continuity of many, many years.
There is a threshold in software complexity after which one really needs lots of organized testing and fixing.
As a private user, if I fail the firmware update on my shiny plane, it's all on me. Unless there was some weird insurance to cover the costs.
Civilization is characterized by specialization of people. I cherish the notion that hardware and software should be based on open standards. For day to day work, I just want my gear to work. If it fails, I certainly do not have the time to dig in to the software layer because I have a work, children, housekeeping duties, and a bunch of art projects and higher level concepts I want to focus on.
"if you have complete visibility into what your machine is doing and have the ability to modify it at will you can avoid all manner of failure scenarios that are essentially unrecoverable in the black box scenario."
You presume all software is trivially simple. I can tell you, it is not. A large category of software requires years of specialization to actually grok what is happening.
As an extreme example if I owned a plane I would not like to hack it's software under any circumstance unless I were a professional aeronautics professional, and probably not even then.