News for hackers. Stuff that matters.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @06:45AM (#22297729)
It sure doesn't look good to have two software issues very late in the product development cycle, on a live test. While the first one would have been averted by a human crew, the second one would not be so benign.

Software is hard. Embedded software doubly so, but... Come on... This should not be happening to Boeing.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by sho_hn on Tuesday February 11, 2020 @06:54AM (#22297766)
One of the takeaways of the MAX debate maybe was "yes, this should be happening to Boeing" as a result of specific steps to weaken the engineering leadership in the company from the top down. If that's the case we can only hope it makes for the case study.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @07:36AM (#22297949)
Kind of yes, but this is a different part of Boeing. If I were one of the astronauts who'll fly Starliner, eventually, I'd not be nearly as polite as the formal communications suggest. I'd probably be reading code, sitting next to an engineer, and asking them to explain to me what the code does.

The thruster map error should have been detected in simulation. If it wasn't, the simulator is also faulty.

I'll assume the thruster map was only slightly wrong and that whatever simulator runs they did with it were very subtly off.

Still... What a mess...

Re:Boeing Starliner's flight's flaws show “fundamental probl... by robkop on Tuesday February 11, 2020 @09:05AM (#22298536)
If you listen to the NASA update [1] you'll hear the audit team state that if Boeing was following the procedures that they stated they were doing they would have found the flaw at 4 separate places. Let's just say the auditors tone of voice says it all.

[1]: https://www.youtube.com/watch?v=hxQ_ZV26E04

Re:Boeing Starliner's flight's flaws show “fundamental probl... by DennisP on Tuesday February 11, 2020 @02:05PM (#22301501)
That's an hour-long video, could you tell us where the auditors said that?
Re:Boeing Starliner's flight's flaws show “fundamental probl... by homonculus1 on Tuesday February 11, 2020 @03:53PM (#22302721)
It starts right at the 47 minute mark, and the whole segment starts at 44:45
Re:Boeing Starliner's flight's flaws show “fundamental probl... by bumby on Friday February 21, 2020 @09:09AM (#22383161)
There are specific requirements to assure the fidelity of simulators so this still should have been caught if the design process was robust enough I think
Re:Boeing Starliner's flight's flaws show “fundamental probl... by MaupitiBlue on Tuesday February 11, 2020 @08:25AM (#22298249)
> specific steps to weaken the engineering leadership

What in the world are you talking about? There’s been no reporting of anything like that. The MAX engineers failed all by themselves. No management told them to design MCAS in a brain dead fashion. Nobody “weakened” engineering leadership.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by toss1 on Tuesday February 11, 2020 @08:39AM (#22298329)
When management makes statements like "we don't need [to pay for] senior engineers here, these are mature products... ", and clearly putting finance first among corporate goals, the natural result is that engineering dies a death by a thousand cuts...
Re:Boeing Starliner's flight's flaws show “fundamental probl... by MaupitiBlue on Tuesday February 11, 2020 @09:27AM (#22298702)
> When management makes statements like "we don't need [to pay for] senior engineers here

The problem is that Boeing management never said that. Moreover, the fired CEO was an engineer.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by TheOtherHobbes on Tuesday February 11, 2020 @09:32AM (#22298739)
The important word being "was".

As an engineer, he was accountable to reality. As a CEO, he was accountable to the board and the financials.

Guess which won.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by Brave-Steak on Tuesday February 11, 2020 @09:45AM (#22298838)
Muilenberg became CEO in 2015. The McDouglas thing happened in '97. He inherited an existing management structure that was already primarily interested in profit over all other concerns, especially engineering.

Moreover, the 737 Max program ran from 2011-2015. The CEO at the time was an MBA.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by throwawayjava on Tuesday February 11, 2020 @07:26AM (#22297898)
> Software is hard

Traditional engineering companies don't respect software. You can tell by the fact that they pay well below market rates.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by jillesvangurp on Tuesday February 11, 2020 @07:53AM (#22298049)
I think maybe the difference here is the fact that SpaceX probably treats software as a first class citizen; just like Tesla does, and is likely to have a modern view on how to build and evolve software. Additionally, software is a key component for them that is very much determining the abilities of the hardware. Tesla even ships pure software features that they charge money for. That's unheard off in the car market. And I would expect, Elon Musk to drive a similar strategy in SpaceX.

I suspect Boeing maybe treats software the same way as they've always done, just one of many boxes that need to be ticked in a long waterfall like process. They just throw more people at it hoping it gets done on whatever time line they imagine it needs to be done. They are likely to be very conservative with respect to tools, technologies, and practices (i.e. not much has changed there in decades).

The fact that SpaceX is able to come up with a new design for Starship and can then get it and the software working for several successful hovers & hops in around 9 months, tells me they are able to iterate quickly and adapt to quite major design changes. They launch a lot of rockets and they seem to be learning a lot every time they do this. I don't see how they'd be able to do this without very solid software development practices. I'd expect lots of CI & test harnasses, probably lots of emulators and other tools, and I have a hunch they probably use a lot of static code analysis etc. to prevent preventable bugs from happening.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by vonmoltke on Tuesday February 11, 2020 @08:52AM (#22298435)
It also helps that there is no union stopping SpaceX from driving it's engineers and technicians to tens of hours of unpaid overtime.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by madamelic on Tuesday February 11, 2020 @09:17AM (#22298619)
I really don't understand this line of thinking.

Any engineer or tech at SpaceX is likely worth their weight in gold at any other company due to the company's perceived high bar. Additionally, I don't hear complaints about their pay either, which is likely extremely good.

No one is locking them into cages and telling them they can't leave. I would bet most everyone who is there strongly believes in the mission.

EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.

https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries...

Re:Boeing Starliner's flight's flaws show “fundamental probl... by vonmoltke on Tuesday February 11, 2020 @10:01AM (#22298967)
> No one is locking them into cages and telling them they can't leave. I would bet most everyone who is there strongly believes in the mission.

That's true for a lot of other companies where people here have advocated the employees would benefit from unionization.

For the record, I am conflicted on engineer unionization, and my original statement was intended to be neutral.

As to this specific example, on one hand I understand the drive that comes from having a real mission (as opposed to an MBA-concocted "mission") and how a team who share that drive can execute with amazing speed; conversely, I have seen how poorly teams of clock-punchers putting in the bare minimum can perform.

On the other hand, I have also seen "the mission" used as a means to guilt trip people into overwork or accepting substandard compensation.

Of the two, I lean towards the SpaceX approach being the better one. In the absolute I don't particularly like either of them.

> EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.

> https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries....

Those salaries look to be in line with what equivalent engineers at Raytheon, Lockheed, and Boeing are making in those same locations, maybe a bit higher. Thus, if the WLB reports from SpaceX are to be believed, their per-hour pay is much lower. It's hard to tell, though, without a proper breakdown by years of experience. Also, there are certainly teams within those companies where the engineers put in just as much time and effort as SpaceX engineers. Boeing is the only one I am aware of that has unionized engineers, though.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by 0xffff2 on Tuesday February 11, 2020 @06:36PM (#22304161)
Eh, I wouldn't say that there's a perceived high bar at SpaceX or Tesla. At this point they're mostly known (within my corner of the aerospace world anyway) for working their employees to the bone. Very few people I know who have worked for SpaceX have lasted more than a year or two there.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @01:32PM (#22301114)
Logic says that would negatively impact the quality of the software. If you are working 50+ hours a week in software development, a lot of what you do will be done when you are too tired to properly reason about what you are doing.

Not that it's impossible, but you'll need tooling that prevents you from making the mistakes you'll unavoidably make when tired.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by marta_morena on Tuesday February 11, 2020 @01:36PM (#22301180)
Sorry but anyone who has worked in a professional high-tech environment should be aware of that an 8 hour days is not productive. You NEED to spend you entire day on this, you mind always needs to be on it to work on that level.

Einstein would certainly not have solved relativity by saying "Uh I guess from 9 to 5 I will work on this with a one hour break in between and then I am heading off to the fun stuff called leisure time".

No... Only if you think about these problems day in and day out you can work at in the top percentiles and this is the kind of people that get hired there.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by perl4ever on Tuesday February 11, 2020 @08:49PM (#22304993)
This is getting cause and effect totally screwed up.

People want to work on cool stuff (in the sense of rockets or in the sense of making the world better, or both). They will accept lower pay (per hour) to do that. Therefore, working on cool stuff demands longer hours and lower pay compared to less glamorous things. It's not an inherent requirement of the field; it's a side effect of human preferences.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by kevin_thibedeau on Tuesday February 11, 2020 @11:07AM (#22299501)
Engineering companies have capital expenses that eclipse what a software shop has to deal with. The rock stars can't demand such a large chunk of revenue.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @01:34PM (#22301152)
Rock stars are neither scalable nor repeatable. Engineering is about repeatability and predictability. Aerospace is that, squared.

I don't want a rock star anywhere near the software that operates a complex machine where a malfunction can kill people.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by chroem- on Tuesday February 11, 2020 @07:59AM (#22298087)
Everything is hard, but software is a lot more vocal about it because it facilitates the modes in which we communicate
Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @07:37AM (#22297957)
So, I guess we should start calling it the Starliner MAX...
Re:Boeing Starliner's flight's flaws show “fundamental probl... by marsokod on Tuesday February 11, 2020 @07:09PM (#22304390)
I am just wondering if all these issues are not linked to a same error that would be: we loaded an old version of the software on the spacecraft.

It can be quite common in the space industry to load non-flight or old versions late in the AIT phase in order to test on other modules, or even just to officially close old actions. But before delivering everything we would check all the versions, and make sure that everything is as expected and most importantly, as described. If that step was screwed up, it is bad news for the QA, but that would explain why the errors could be found and fixed quickly: it meant just going through all the non-conformances and change logs and understand what was different between the version used, and the version that should have been used.

[dead]
Re:Boeing Starliner's flight's flaws show “fundamental probl... by WhipGhostrider on Tuesday February 11, 2020 @12:37PM (#22300443)
I have recently worked on software testing for a roughly analagous project to Starliner. The answer is thorough simulations and ground testing. Flight code should have essentially 100% coverage in unit tests as well.

You can do a pretty decent simulation of a space mission on a powerful cluster. You then plug in your simulated spacecraft and run the real flight software. Reentry is easily the most critical phase of flight for a crewed spacecraft, and they should have absolutely tested and caught this flaw in simulation. Ground testing could have also checked on the actual spacecraft that the correct valves were actuated when commanded to.

If I were to speculate, this suggests a pretty large disconnect between the software and hardware engineering teams. The software team should have been able to identify that the wrong valves were being actuated.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by marcinzm on Tuesday February 11, 2020 @08:18AM (#22298203)
Starliner has been in full development for five years and in preliminary development for 9+ years.

>How can that even be "reviewed" in any way but superficially in a time frame less than 5 years?

If it can be written then it can be reviewed, you just need to pay for more engineers to do that.

Re:Boeing Starliner's flight's flaws show “fundamental probl... by sudoaza on Tuesday February 11, 2020 @08:12AM (#22298167)
just checked the project i'm working on, its close to 1 million lines of code without considering dependencies and we are just 3 devs, not that hard, if they have a decent team (as i would expect), a thorough test environment and processes focused on security this should not happen. They are probably rushing or outsourcing parts of it like everything else they do.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by aforwardslash on Tuesday February 11, 2020 @11:09AM (#22299524)
Well, it really depends on what the project does. Most general purpose programming has looser requirements (both on specification and implementation) than critical projects. I have the sense you're comparing apples and oranges. Have a look at ex. MISRA https://en.wikipedia.org/wiki/MISRA_C to have an idea of the strictness applied. Hint: if you're using any methodology from the "agile" umbrella, it's not even in the same league. Specification and verification processes are arguably the most time-consuming part, not maintaining the code by itself.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by sudoaza on Tuesday February 11, 2020 @06:15PM (#22304009)
Checked it, it's nice to have a standard linter, but then even at web development we use those. I know C/C++, embed systems and orbital dynamics will be harder but I would expect them to have the whole mission tested and dry-ran before, at the very least. I'd bet they were just rushed to deliver and focused on a working features checklist instead of thorough testing.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by 0xffff2 on Tuesday February 11, 2020 @06:41PM (#22304198)
I you think MISRA-C is just "a standard linter" I don't think you've really grasped what it requires. MISRA-C compliant code requires a pretty fundamental shift in thinking.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by sudoaza on Friday February 14, 2020 @03:47PM (#22330242)
There are some good common sense recommendations on the general development and then there are basically linter rules. The rules may be more restrictive than others but makes sense for a language such as C and the application is being used. But all this rules can be implemented by a linter and a short checklist. http://caxapa.ru/thumbs/468328/misra-c-2004.pdf
Re:Boeing Starliner's flight's flaws show “fundamental probl... by 0xffff2 on Tuesday February 11, 2020 @06:39PM (#22304182)
Agile has nothing to do with this. I don't want to dox myself, but I currently work on a spacecraft flight software team and we use a lot of agile principles in our development process.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by andruby on Tuesday February 11, 2020 @09:01AM (#22298504)
Are you saying your team of 3 devs are maintaining 1M lines of code? Roughly how many years and devs worked on it in total?

I guess I’m spoiled working on a (mostly) Ruby project with ~100K-ish lines of code with a team of 25

Re:Boeing Starliner's flight's flaws show “fundamental probl... by sudoaza on Tuesday February 11, 2020 @05:46PM (#22303812)
More than 5 years for sure, the thing is ancient. Quite big Ruby CRM with some IOT on the side. How did you find 25 Ruby devs? I though we were a rarity.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by andruby on Wednesday February 12, 2020 @05:44AM (#22307712)
We're even more rare in Belgium or Europe (vs USA). That's why we made the engineering team remote from day 1.

We're about to start our next hiring spree in case you're looking for a new challenge ;)

Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @07:39AM (#22297968)
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
Re:Boeing Starliner's flight's flaws show “fundamental probl... by ceejayoz on Tuesday February 11, 2020 @07:53AM (#22298046)
Weird to cite Feynman here, in this context.

https://en.wikipedia.org/wiki/Rogers_Commission_Report

Re:Boeing Starliner's flight's flaws show “fundamental probl... by rbanffy on Tuesday February 11, 2020 @01:27PM (#22301038)
It was deliberate. This is how his personal observations on the reliability of the space shuttle end.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by perl4ever on Tuesday February 11, 2020 @08:52PM (#22305015)
At that time, he noted that the Space Shuttle software was the one area that didn't show deficiencies.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by mkl on Tuesday February 11, 2020 @07:42AM (#22297989)
With enough developers. Why do you see that as a problem? If the code is well structured, most parts of it can be reviewed and tested in parallel.
Re:Boeing Starliner's flight's flaws show “fundamental probl... by travisoneill1 on Tuesday February 11, 2020 @07:55AM (#22298060)
I would bet that the truly critical parts of the software are a fraction of that million lines.