I have recently worked on software testing for a roughly analagous project to Starliner. The answer is thorough simulations and ground testing. Flight code should have essentially 100% coverage in unit tests as well.

You can do a pretty decent simulation of a space mission on a powerful cluster. You then plug in your simulated spacecraft and run the real flight software. Reentry is easily the most critical phase of flight for a crewed spacecraft, and they should have absolutely tested and caught this flaw in simulation. Ground testing could have also checked on the actual spacecraft that the correct valves were actuated when commanded to.

If I were to speculate, this suggests a pretty large disconnect between the software and hardware engineering teams. The software team should have been able to identify that the wrong valves were being actuated.

Starliner has been in full development for five years and in preliminary development for 9+ years.

>How can that even be "reviewed" in any way but superficially in a time frame less than 5 years?

If it can be written then it can be reviewed, you just need to pay for more engineers to do that.

just checked the project i'm working on, its close to 1 million lines of code without considering dependencies and we are just 3 devs, not that hard, if they have a decent team (as i would expect), a thorough test environment and processes focused on security this should not happen. They are probably rushing or outsourcing parts of it like everything else they do.

Well, it really depends on what the project does. Most general purpose programming has looser requirements (both on specification and implementation) than critical projects. I have the sense you're comparing apples and oranges. Have a look at ex. MISRA https://en.wikipedia.org/wiki/MISRA_C to have an idea of the strictness applied. Hint: if you're using any methodology from the "agile" umbrella, it's not even in the same league. Specification and verification processes are arguably the most time-consuming part, not maintaining the code by itself.

Checked it, it's nice to have a standard linter, but then even at web development we use those. I know C/C++, embed systems and orbital dynamics will be harder but I would expect them to have the whole mission tested and dry-ran before, at the very least. I'd bet they were just rushed to deliver and focused on a working features checklist instead of thorough testing.

I you think MISRA-C is just "a standard linter" I don't think you've really grasped what it requires. MISRA-C compliant code requires a pretty fundamental shift in thinking.

There are some good common sense recommendations on the general development and then there are basically linter rules. The rules may be more restrictive than others but makes sense for a language such as C and the application is being used. But all this rules can be implemented by a linter and a short checklist. http://caxapa.ru/thumbs/468328/misra-c-2004.pdf

Agile has nothing to do with this. I don't want to dox myself, but I currently work on a spacecraft flight software team and we use a lot of agile principles in our development process.

Are you saying your team of 3 devs are maintaining 1M lines of code? Roughly how many years and devs worked on it in total?

I guess I’m spoiled working on a (mostly) Ruby project with ~100K-ish lines of code with a team of 25

More than 5 years for sure, the thing is ancient. Quite big Ruby CRM with some IOT on the side. How did you find 25 Ruby devs? I though we were a rarity.

We're even more rare in Belgium or Europe (vs USA). That's why we made the engineering team remote from day 1.

We're about to start our next hiring spree in case you're looking for a new challenge ;)

"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman

Weird to cite Feynman here, in this context.

https://en.wikipedia.org/wiki/Rogers_Commission_Report

It was deliberate. This is how his personal observations on the reliability of the space shuttle end.

At that time, he noted that the Space Shuttle software was the one area that didn't show deficiencies.

With enough developers. Why do you see that as a problem? If the code is well structured, most parts of it can be reviewed and tested in parallel.

I would bet that the truly critical parts of the software are a fraction of that million lines.