News for hackers. Stuff that matters.
Re:Boeing Engineers Lost Control of the Company by dmix on Sunday October 20, 2019 @11:40AM (#21304793)
Has anyone considered that the MCAS bug was just simply nasty and difficult to detect in their simulator?

It’s easy to look back with retrospect and make sweeping judgements about the entire company and the FAA.

But fundamentally this was all over, as the author mentions, a single software bug and a lack of backup sensors to offset the risk.

In addition it was the technical pilots who asked for the stick to be harder than it was, which added to the difficulty in that unique situation.

The MAX was otherwise a very safe plane was it not?

There will always be limits to how effective regulatory oversight can be.

Re:Boeing Engineers Lost Control of the Company by sqldba on Sunday October 20, 2019 @11:50AM (#21304872)
The article is not a complete overview of the situation. For that we go to Wiki.

"Boeing considered MCAS part of the flight control system, and elected to not describe it in the flight manual or in training materials."

It... "could not be overridden by pilot resistance against the control wheel as on previous versions of the 737."

"Following the Lion Air accident, Boeing issued an Operations Manual Bulletin (OMB)[13] which outlined the many indications and effects resulting from erroneous AOA data and provided instructions to turn off the motorized trim system for the remainder of the flight, and trim manually instead. Until Boeing supplemented the manuals[14] and training, pilots were unaware of the existence of MCAS due to its omission from the crew manual and no coverage in training.[12]"

It wasn't just "a software bug". It wasn't just culture cutting corners. Even after the first crash they continued to cover up that MCAS was the cause. After the second crash they still refused and it wasn't until airlines started grounding planes that they caved in.

There's a lot more than regulatory oversight could have done. We haven't reached the limits in the slightest.

Re:Boeing Engineers Lost Control of the Company by angry_octet on Sunday October 20, 2019 @12:43PM (#21305245)
Agreed. There is a key part of the regulatory system which has not happened yet, which is deterrence. People need to suffer serious personal consequences for dereliction and cover-up, including criminal liability. In fact, the post Lion crash cover-up should perhaps be punished more severely than the decision to leave known risks (AoA sensor failure, inability to override) untreated.
Re:Boeing Engineers Lost Control of the Company by CydeWeys on Sunday October 20, 2019 @12:55PM (#21305338)
"The MAX was otherwise a very safe plane was it not?"

Huh? The MAX was the most unsafe major new plane in ... something like decades. You can't make a statement excluding all the bad and then trying to say it's good. The bad is exactly what you're getting at here.

And even apart from that, the 737 is an obsolete, decades-old design. The only reason they're still making new iterations of it is because of type ratings requirements; if you actually wanted to make a great plane, you'd never base it on the 737. It has ancient analog gauge displays (which are now simulated digitally), doesn't use electronic flight controls, is too low to the ground leading to increased potential of foreign object debris into the engines, and as a consequence of this the newer more efficient jet engines don't even fit underneath the wings properly and hence had to be shifted forward into an unstable position that necessitated MCAS.

There's no two ways about it, there's nothing good about the 737 MAX, even apart from all the crashes.

Re:Boeing Engineers Lost Control of the Company by klyrs on Sunday October 20, 2019 @12:05PM (#21304969)
> Has anyone considered that the MCAS bug was just simply nasty and difficult to detect in their simulator?

That was the assumption a year ago. Investigation has turned up tons of evidence against your null hypothesis. Crucially, this behavior was observed in the simulator and reported up the line.

> ... and a lack of backup sensors to offset the risk.

Not when multiple red flags were raised internally, and the backup sensors were sold as a premium upgrade.

> There will always be limits to how effective regulatory oversight can be.

Especially when we allow corporations to do their own regulatory "oversight"

Re:Boeing Engineers Lost Control of the Company by acomjean on Sunday October 20, 2019 @11:52AM (#21304883)
Its not that it was just a difficult bug.

Its that politics got in the way of doing the right thing, which is declare the airplane "new" and re-train the pilots (vs a plain modification). Boeing didn't want to do that as customers didn't want the retraining expense so they used lobbying and cut the corner.

Re:Boeing Engineers Lost Control of the Company by binky on Sunday October 20, 2019 @11:45AM (#21304833)
The bug is a symptom of bad architecture. Bad architecture is an outcome of taming loud engineers in sake of short term business outcomes.
Re:Boeing Engineers Lost Control of the Company by linuxftw on Sunday October 20, 2019 @03:32PM (#21306476)
Pretty much. If something is theoretically possible it needs to be defended against. I'm sure there's no shortage of people that could have looked at the requirements and figured out how to make it break, but Boeing outsourced.

I'm hoping we get more details so we can help build a text-book case against software outsourcing.

Re:Boeing Engineers Lost Control of the Company by privateSFacct on Sunday October 20, 2019 @11:59AM (#21304942)
It also relied on a the pilot as a critical control - which is why there was no real automated backup or redundancy.

Not unreasonable, doing stab trim cutout is a memory item for most operating manuals - pilots were already the circuit breaker for other stab trim issues.

Automation working - great. Maintenance failures? Provide error messages (which occured here). Maintenance issues not resolved and automation misbehevior? Stab trim cutout memory item.

Despite all the claims that the stab trim cutout did not work all real evidence is that it did. Other issues around pressure on flight surfaces not explored properly making recovery in overspeed and other conditions very hard.

Badly designed system? Definitely. Brittle when facing poor maintenance? Clearly! Not taking into account broad pilot population. Of course. Definitely should be fixed, and frankly automation probably will be increased to decrease assumptions around pilot behavior which can be variable.

This accident was the result not only of this bug but a chain of issues. That's why flying in the US is pretty safe, the chain from procurement to maintenance to pilot training is strong so compensates for poor elements.

If you follow SAA technical for example there is evidence overseas that bogus parts are making it into plane repairs, with predictably bad results. We are not seeing that in the US (yet), though if inspections get soft I'm sure it would show up here too. Those bogus repairs will mean that some automation will break and the plane may not behave properly in ways that could have been designed around with better assumptions.

My prediction? We are going to get iphone style authentication / serial number chips in key plane parts with electrical connections and the plane will eventually refuse to fly without the right parts - just like your coffee maker and iphone. Right now it is trust based.

Re:Boeing Engineers Lost Control of the Company by largbae on Sunday October 20, 2019 @02:09PM (#21305868)
s/This accident/These accidents/

As you'll find in this article below, Boeing specifically removed the yoke-back function, which is the natural pilot input to nose up the plane. They also specifically for the MAX removed the separate switch for automatic stab trim assistance, so you have to turn off the motorized function entirely to turn off automated inputs including MCAS. The Lion Air crew could be heard on the radio saying that they did disable the stab trim, but in this situation the force required on the manual trim wheel was beyond the pilots' strength. The pilots turned the electronic stab trim back on in hopes of using the powered system to get back up, after which the undocumented and unstoppable MCAS nosed down the plane again, killing them and their passengers.

https://www.seattletimes.com/business/boeing-aerospace/boein...

Your post reads like a sophisticated attempt to blame the victims, or at least the pilots and ground crews of the victims. These crashes were not caused by poor pilots, but rather secrets on top of secrets, all by Boeing, all for money.

The MAX had new aerodynamic characteristics, but that needed to be a secret.

So MCAS was added to the electronic trim assistance to hide the new characteristics, and the switch to disable it was removed to protect the secret.

A faulty AOA sensor was now able to bring down the plane, but in order to keep the secret the single AOA option was still available and sold. Maintenance crews regardless of quality would not have been aware of the criticality of the AOA's secret role as a single point of failure for the whole plane.

When the first one crashed, secrets covered it up.

When the second one crashed, it still took all of those "ppor maintenance", "broad quality" non-US airlines and regulators to bring a stop to it.

And apparently the secret-keepers are still trying to hide.

Please don't blame dead pilots for this.

Re:Boeing Engineers Lost Control of the Company by salawat on Sunday October 20, 2019 @07:39PM (#21308113)
One more thing, the flight computer was eventually found to be a single point of failure after being subjected to testing that should have been done in the first place.

https://www.seattletimes.com/business/boeing-aerospace/newly...

Another regulatory no-no that wasn't brought up until the certified aircraft had been out in the wild long enough to kill people.

Re:Boeing Engineers Lost Control of the Company by privateSFacct on Sunday October 20, 2019 @08:35PM (#21308378)
Is this the five independent bit flips in worst possible order and combination from cosmic rays failure?

First -> if this is your worst failure mode (ie, worse than pilot error, mechanical fatigue and failure, engine failures, fueling issues, crazy passengers etc) then you have a plane and overall safety system that is incredible.

Second -> this wild set of events again does not crash the plane - though it would put the pilots in a bad spot similar to the existing MCAS issues.

Re:Boeing Engineers Lost Control of the Company by salawat on Sunday October 20, 2019 @10:05PM (#21308820)
>First -> if this is your worst failure mode (ie, worse than pilot error, mechanical fatigue and failure, engine failures, fueling issues, crazy passengers etc) then you have a plane and overall safety system that is incredible.

Or would be, if it didn't violate design constraints that demand no single point of failure can have the capability to result in loss of the aircraft. I.e. components with failure profiles of catastrophic severity MUST have redundancy. You test the least likely events to ensure you have sufficient fault tolerance for that very reason.

>Second -> this wild set of events again does not crash the plane - though it would put the pilots in a bad spot similar to the existing MCAS issues.

Actually, one of the three test pilots did in fact lose the plane in the simulator. Hence the reclassification of the flight computer as a component with a catastrophic failure potential.

Look, I'm about the last person you're going to get to buy into the awesome safety of an aircraft that can't even pass a textbook test case for computing in adverse environments. If the bloody thing was so safe and high quality, it wouldn't have been able to suffer that induced failure in the first place. It would have been designed out. The fact Boeing was caught off guard by that failure means they weren't looking very hard, and were banking that no one else would either. Which is generally a really bad assumption to make, because Murphy finds a way. Every time.

Re:Boeing Engineers Lost Control of the Company by privateSFacct on Sunday October 20, 2019 @11:31PM (#21309133)
There's lots of potential for failure that the planes allow when flying.

Some quick examples - they could have sensors around gas quality but do not - they rely on the folks providing the avgas to have good fuel. They could do built in weight and balance calcs with sensors - they do not, they rely on poor estimates for that. etc.

Boeing in particular ALLOWS pilots to fly planes outside of most flight envelope limits. They do this in part because historically they didn't trust the computer as much and trusted pilots more.

Airbus by contrast doesn't let you do this unless you take air data totally offline or something crazy to force alternative law. I suspect boeing will be moving towards the airbus model which is probably right. Airbus had arguably less local myopia when designing their planes - ie, future pilot populations come from a big group and making the plane as error proof as possible is best. Airbus had some issues around their automation without pilot override initially but I think those are resolved by now.

If you want a scary video - check out this early A320 flying - first fly by wire / autothrottle:

https://www.youtube.com/watch?v=YAg-WauGrLU

Can't have been a good feeling as that pilot. Note that the A320 is now considered a very safe plane.

Aircraft have evolved from systems with almost no built in redundancy or safety. The design of the 737 is pretty old - 1967 or so. Let that sink in for a bit, consider what type of tech they had back in. This is not a 777 generation plane or even A320.

Re:Boeing Engineers Lost Control of the Company by privateSFacct on Sunday October 20, 2019 @08:19PM (#21308282)
Stab trim cutout has been the response item for a long time. Before the stab trim was turned on again the plane was climbing.

"unstoppable MCAS"

* Leave stab trim cut

* Hold electric trim up

* Control airspeed

* etc...

AoA senor is not a single point of failure. The 737 is not a fully automatic / flight protected plane. ie, the plane will continue flying with very little sensor input at all - you can even dispatch with problems with a surprising amount of equipment under the MEL.

Re:Boeing Engineers Lost Control of the Company by largbae on Sunday October 20, 2019 @09:21PM (#21308609)
Thank goodness it's so safe! I am glad we are only talking about theoretical crazy combinations of events that can lead to a crash...

You brought up faulty replacement parts yourself as a contributing cause, but when challenged you switch to methods to stop the unstoppable, undocumented MCAS. To which faulty part were you referring? The unimportant, non-SPOF AOA sensor?

Re:Boeing Engineers Lost Control of the Company by privateSFacct on Sunday October 20, 2019 @11:18PM (#21309077)
The cosmic ray bit-flip of the 5 independent critical bits is very unlikely. We are being told right here on HN and in the news that this means the plane can't fly.

In the US there were close calls, but fewer. If you beleive in the chain of safety better maintenance and pilot response may have saved the day a few times when faced with the MCAS problems. The safety reports from US pilots certainly sound scary (nose down on takeoff GPWS alerts etc).

I brought up maintenance as a contributing cause in the 737MAX crashes. I didn't realize I had been challenged on this. I would really encourage you to read some of the source reporting.

The FAA had a safety notice on this precise topic

"FAA has determined it is necessary to advise operators of the importance of performing proper operations and maintenance on AOA sensors”

This is the detail on the previous lionair flight before the crash flight - from wikipedia to save time:

"it had problems maintaining a constant altitude, with passengers stating that it was like "a roller-coaster ride".[109] The chief executive officer of Lion Air, Edward Sirait, said the aircraft had a "technical issue" on Sunday night, but this had been addressed in accordance with maintenance manuals issued by the manufacturer. Engineers had declared that the aircraft was ready for takeoff on the morning of the accident.[110][111] A later report claimed that a third pilot was on the flight to Jakarta and told the crew to cut power to the stabilizer trim motors which fixed the problem. This method is a standard memory item in the 737 checklist.[112] Subsequently, the National Transportation Safety Committee confirmed the presence of an off-duty Boeing 737 MAX 8 qualified pilot in the cockpit but did not confirm the role of the pilot in fixing the problem.[113]

Even worse is the history before that flight - REPEATED problems with the senor.

We are also now hearing that evidence submitted by lion air supposedly showing the proper fix for the sensor may have been faked.

https://www.seattletimes.com/business/questions-raised-about...

In terms of faulty replacement parts - my point was issues occur overseas that do not seem to happen within the US MROs. I'll do a quick recent example:

https://www.pressreader.com/south-africa/sunday-times-1107/2...

This follows the ex-president of south africa appointing a school teacher to head SAA (who also ran his private foundation and rumors around their relationship existed).

I'm not sure why you claim AoA is unimportant. In the chain of safety you want every advantage you can get, the more accurate data the better. So you want the AoA working. This reduces risks from MCAS. You are correct - AoA is not a single point of failure system (system is not working / plane falls from sky). There was a poster here claiming that it was.

Re:Boeing Engineers Lost Control of the Company by largbae on Monday October 21, 2019 @06:24AM (#21310686)
I've got your point, and I think that knowing what we know now perhaps the planes could have been saved. I'm still not for blaming the pilots though: Would you have wanted to be the captain of a max while MCAS operated against a faulty AoA without having any way of knowing that MCAS existed or of the handling/stall risk that led to its creation?
Re:Boeing Engineers Lost Control of the Company by privateSFacct on Tuesday October 22, 2019 @05:57AM (#21321876)
Obviously not. There is a difference between blaming the pilots - I am not, and saying the plane was "unrecoverable" the system was "unstoppable".

Luckily if flying in the US the chance of the AoA being faulty in the first place would be a less I would argue than flying for lionair. My co-pilot might be quick to help. I might be quicker to get off automation and manually fly / trim.

Some of the self reports from the US specifically credit co-pilot assistance in averting a problem. See this one during takeoff - worst time to be heading towards ground at 1,500 FPM!

Within two to three seconds the aircraft pitched nose down bringing the VSI to approximately 1,200 to 1,500 FPM. I called "descending" just prior to the GPWS sounding "don't sink, don't sink."

Other side:

"PM's (co-pilot) callout on "descending" was particularly quick and welcome as I was just coming back to my display after looking away. System and procedures coupled with CRM (Resource Management) trapped and mitigated issue.

Synopsis

B737MAX Captain reported an autopilot anomaly in which led to an undesired brief nose down situation."

This was a full GPWS don't sink alert situation - very cleanly handled - the co-pilot BEAT the automated alert, the recovery was good. Should pilots have to deal with this? No. Did chain of safety work here? Yes - though closer to edge than one would like.

My own impression - US pilots much quicker to get off the automation / don't trust it as much / have more comfort or experience hand flying statistically maybe.

Re:Boeing Engineers Lost Control of the Company by selimthegrim on Sunday October 20, 2019 @09:02PM (#21308509)
As Sullenberger said in his NYT reply to Langewiesche, blaming the pilots is out of line. I don't see anyone blaming that CHP officer who died in one of those unintentionally accelerated Toyotas because he in his panic didn't think to shut off the engine maybe for fear of losing power steering or most of the brake boost?
Re:Boeing Engineers Lost Control of the Company by jorprp93j on Sunday October 20, 2019 @12:06PM (#21304978)
Did you see the article where pilots testing the MCAS software said it was a mess? Years before they shipped it?
Re:Boeing Engineers Lost Control of the Company by golemotron on Sunday October 20, 2019 @11:44AM (#21304824)
It was simply bad systems engineering. Using software to compensate for hardware like that is like coding "subtract one" when two plus two yields five.

In software this hack is explicit, so perhaps Boeing engineers can be forgiven for not seeing it when it straddles hardware and software.

Re:Boeing Engineers Lost Control of the Company by sambull on Sunday October 20, 2019 @11:58AM (#21304936)
No its not to the standards of a modern design. It's one of the primary reasons they grandfathered the airframe in, not having to address safety concerns like their doors.
Re:Boeing Engineers Lost Control of the Company by meristem on Sunday October 20, 2019 @12:04PM (#21304965)
They grandfathered the frame--and added MCAS-- to avoid a new certificate and added cost/time to their customers. What you point out is one of many consequences, not the reason.

(Aircraft doors are interesting on their own as well, due to the assumptions that drive design, testing, and certification.)