Boeing used to be an engineering-first company. HQ was at Boeing's own airport near Seattle. Then they got new management and moved corporate HQ to Chicago.
In particular when you are doing something for the first time, uncertainty creates a space for a safety culture to operate.
Design, process and safety emphasis lead to operational success.
Over time the original designers and engineers leave, new staff join. Field experience shows that most of the safety problems that were anticipated in design never happen. The organisation starts to become complacent.
All the safety features and procedures start to be considered, psychologically, as a “safety margin” or a kind of budget that can now be spent by taking more risks. Triple or quadruple redundancy just starts looking like extra weight.
Within the org, pressures to lower costs and deliver faster start to win out...
https://flightsafety.org/asw-article/normalization-of-devian...
This lifecycle pattern can be found at other organizational scales, including countries, e.g. soldier -> law -> artist -> merchant -> warlord -> soldier ..
> I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study Mathematicks and Philosophy, Geography, natural History, Naval Architecture, navigation, Commerce and Agriculture, in order to give their Children a right to study Painting, Poetry, Musick, Architecture, Statuary, Tapestry and Porcelaine.
And the military A400M -- 4 engine turbprop -- https://www.youtube.com/watch?v=voK2o62KpqI -- quite amazing to see when you are used to those tiny regional turboprops.
Impressive performance with the plane empty and probably only an hour or two of fuel.
The 707 was pretty similar to the 737...same main fuselage dimensions, similar pax capacity, etc.
I believe they fired him for it - though then hired him back after 24 hours and the realisation that it was a stunning piece of advertising.
https://www.seattletimes.com/seattle-news/60-years-ago-the-f...
What essentially was a very engineering driven culture was replaced by finance and marketing geeks that very much moved away from this approach (and moved Boeing's headquarters to Chicago in 2001 and in the process management away from engineering and construction).
Sadly (and arguably) culminating in the two crashes we read about recently.
> Nowhere in here is there a software problem.
I worked on the design of the 757 stab trim system, am trained in aerospace engineering, and am a programmer. I'm not a pilot. I am not explicitly familiar with the 737 stab trim system.
It is indeed at least partially a software problem. I had thought of two possible improvements: one is to limit the authority of the MCAS's commands to the trim, and the other is to not issue further commands if the pilot fights the trim commands. I read today that Boeing's proposed software fix includes both of those.
This is a pretty bold statement. The spec includes many non-software related features.
I really question what the hell happened here: - Did they just pencil whip the FMEA on this or what? (failure mode and effects analysis)
- What happened with the Hardware-in-the-loop flight simulation when they tested the scenario where the AoA sensor givesg spurious data, both high and low (but especially high)? I mean... they did test this, right?
(And yes I am oversimplifying.)
This is simply false. Boeing has spent a lot, but less than General Electric, or the National Association of Realtors, or many (many!) healthcare and pharmaceutical companies.
Source: https://www.opensecrets.org/lobby/top.php?indexType=s
Fair enough, I guess. I think of groups like Blue Cross/Blue Shield or the NAR as representing corporations, but I can see how you'd disagree on a strict reading. I didn't mean to mislead.
> General Electric has fallen off a cliff since 1998.
In 1998, GE spent $7.28m on lobbying. Nearly every year since then, they have spent more than that, generally staying north of $16m per year, with the exception of the last three years.
Your statement is true only for the last three years.
Same source: https://www.opensecrets.org/lobby/clientsum.php?id=D00000012... for GE, https://www.opensecrets.org/lobby/clientsum.php?id=D00000010... for Boeing.
After regulatory capture, it is not clear to me that any subsystem reviews are valid.
MCAS is not the only subsystem revised (or introduced) for the MAX series.
All "FAA" so-called reviews need to be inspected for validity. (Quotes because Boeing not the FAA did the reviews). And recertification will take months.
A phrase comes to mind: "To screw up this bad you need an MBA." Which is not really intended as a dig on MBA's... but rather to indicate that this debacle has the feeling of business decisions overruling sound engineering design.
This seems poor system engineering on multiple levels:
- the whole airframe / engine thrust composite flight characteristic requiring an agressive AoA limiting safety... on a commercial airliner? what?
- The default MCAS relying on a single sensor, which seems to be occasionally quite unreliable, without sensor truth'ing based on other measurements (i.e. dividing ground speed by rate of altitude change to estimate AoA, then also figuring for control surface position feedback.
- The MCAS not having proper situational behavior: i.e. "I'm only 500 feet of the deck... maybe this not a great time to try to tip the nose down too hard... you know, in case that stupid sensor is wrong. Oh... and the pilot is also pulling back on the yoke trying to correct something I'm doing. yeah... my bad, MCAS will chill right now and assume the pilot is NOT trying to fly this thing like a stunt plane 500 feet off the deck".
Without going into a lengthy analysis... you can infer AoA from groundspeed, rate of climb/descent, pitch/yaw/roll, and position feedback of control surfaces. Recent information on these data points can be used to calculate present and near future flight dynamics. The fields of Kalman filtering, adaptive control, and optimal control as applied to aerospace engineering are decades old.
There are many examples where adaptive flight controls can fly an aircraft that has inherently unstable flight dynamics... In fact most modern jet fighter require computer controlled flight controls to maintain stable flight.
Applying these algorithms to an airliner for sensor truth'ing is a cake walk.
* I think you actually can infer situational AoA from ground speed and altitude data: An example you could build a regression model of standard take-off scenarios and using recent groundspeed, altitude change, and the rate of change of (groundspeed/altitude). You could build this off of flight test data and simulation. In practice, there may be no point to doing this because a wealth of additional flight sensor information is available.
At the end of the day the fail-safe is to throw control back to the human. And, then of course pin all the liability onto the human operator when things go wrong.
This was the problem I had with the Lion Air conclusion. Sure, it may be the case that a pilot can potentially override this system with situational awareness and training. However, the pilot didn't create this band-aid pile in the first place.
Wouldn't that depend on how often MCAS is actually needed, and how it handles divergent sensors?
My understanding as a layman is that there are situations where a 737MAX is more likely to stall than a "regular" 737 would, but I haven't seen anything on how often those occur. Are they expected to routinely occur, so a fully functioning MCAS is important for safe operation of the plane?
Or are they supposed to be rare, one in several million events, and the MCAS is there to make handling those events the same as they would be handled in a "regular" 737, to minimize the need for retraining?
If the later, then I'd expect two sensors would be fine, or even just one sensor IF there was a reliable way to tell if it had failed, as long as the MCAS response to a sensor problem is to disable itself and let the pilots know they have to stay out of those "more likely to stall" situations for the rest of the flight.
In computing, we have a principle stated Garbage in, Garbage out. Circuits and electronics don't think. They compute. There is no error checking except that which is specifically designed and implemented into the system.
If you're getting data from a biased sensor with +/- 10 degrees AoA, a 10 degree actual AoA (well within the safe operating envelope) suddenly appears to MCAS as a 20 degree AoA (oh shit territory).
The system therefore engages doing exactly what it was designed to do.
Thus is the crux of the matter. The pilot was flying safely while his AoA sensor was telling a safety system he knew nothing about that he was flying dangerously.
The AoA system on earlier models of aircraft that the MAX was based on were a functional luxury/situational awareness aid. The flyability was not impacted by a horked sensor.
That changed once they had to add a software driven mechanism to keep the flight characteristics similar enough with the old airframe to be able to release the aircraft to 737 trained pilots, and not have to worry about retraining. Maintenance and pilot alike both needed to be aware that the AoA sensor became a safety critical component due to a failure or miscalibration jeopardizing the controllability of the airframe.
If they had gone through a full recert of this airframe, and not an expedited self-certify/grandfathering, these tragedies would have had a much smaller likelihood of occurring due to the increased scrutiny. Props on Brazil for doing their own footwork, and not blindly trusting the FAA's delegation to the manufacturer.
Personally, as a software engineer and quality assurance specialist, if I'd seen anything remotely like this come across my desk, I'd be raising hell, even if it meant yanking someone into a VP/C's office and giving them a dressing down for skimping on a cross-cutting safety critical concern, deadlines be damned.
Is that true with the Ethiopian Airlines crash? I can believe it was true with Lion Air, but that crash was so widely publicized that I'm surprised that any passenger airline captain had not heard of the MCAS system.
I'd not be surprised if there will be another problem with MCAS detected with the ET flight where the pilots had effectively no chance despite knowing of the Lion air crash.
(just speculation of course.)
In systems with dual redundant channels you don't try to determine which one is correct (unless it's obvious, like lost comms to one sensor), you just throw an error and stop trusting both sensors.
But, yes. Most of these systems are not designed to be resilient to that class of issues, and just send control back to the operator.
So perhaps Boeing felt that this didn't really change anything in that regard. They seem to have been proven wrong.
* Control airplane pitch attitude manually with control column and main electric trim as needed
* If the runaway stops, stop.
Well, electric trim input on the yoke will stop MCAS temporarily. Of course if these guys had any experience on an NG they're already used to the computers trimming the plane in a counterintuitive manner via the speed trim system (STS). So not only is MCAS not a runaway trim situation, but pilots flying the NG will get used to the computer trimming the stabilizer "at random".
1: http://www.737ng.co.uk/737-800%20Quick%20Reference%20Handboo...
4 If the runaway continues:
STAB TRIM CUTOUT switches (both) . . CUTOUT
If the runaway continues:
Stabilizer trim wheel . . . . . Grasp and hold
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
5 Stabilizer . . . . . . . . . . . . Trim manually
This is what the Lion Air crew of the flight before the one that crashed did, in response to the same sort of MCAS failure, and completed the flight without further trouble. While Boeing made a serious error in hiding the differences between this variant and its predecessors, it seems that the prior trim runaway procedure does work for this sort of MCAS failure (unless evidence to the contrary comes out of the Ethiopian Airways investigation.)
I think the problem is that the symptoms are different from other 737s, and if you don't KNOW about the MCAS stuff, that's very surprising. So as humans, even though logically the same procedure applies, instinctively you don't think to do it. And given the very short time frames involved, and the margin of error, it's not too surprising that these events have occurred.
Perhaps the need-to-retrain criteria needs to include the following amendment:
If the craft exhibits new symptoms, even if the response to those symptoms are covered correctly be the old operations manual, pilots will need to be retrained to account for the new symptoms.
No, it doesn't and that's the problem. A runaway won't stop when you hit the trim buttons.
Depends on what causes the runaway. If the autopilot goes berserk, the trim buttons will override it (if I recall correctly, it's been a long time).
So while yes, runaway trim is a thing, the circumstances under which an error could happen are substantially different.
For instance, a pilot could check the maintenance log book and see some work or inspection was done on the auto-trim subsystem. This would prime the pilot to be more on the alert for the possibility of trim misbehavior during the flight. Crisis averted, right?
But no pilot would look at an entry for an AoA sensor being off or worked on, and would think, "Boy, better look out for that safety system I never knew existed that could cause a trim runaway because my previously airworthyness agnostic AoA sensor is unreliable."
Flying and complex system diagnosis in time sensitive conditions requires extensive ahead of time mental model creation ahead of time.
If anything, the process of engineering is most difficult in that answering most questions tends to be straightforward (not easy, but straightforward once you know the right methods to apply), while figuring out if you've asked all the right questions is the thing that keeps me awake at night. If you don't ask and dig in until you've answered it fully, it's easy to get blind sided.
I have difficulty believing it could have gone so horribly wrong now, but I can't deny that just from the information available, even if MCAS isn't the root cause of the Ethiopian Air crash, there is some egregious failures in sound practice going on with Boeing to have slipped up this badly.
That's the thing with sound Engineering, when you do it, it just verks. When you don't...
That's a tall order when the manufacturer doesn't inform their customers out of commercial expediency.
You could choose to select the input that results in the least-worst bad outcome. But that may not bring the total risk below the increased risk of utilizing both sensors. The sensors are doing something purposeful, after all.
So it may be logical what they're doing. OTOH, the logic may make sense only in a path dependent sort of way, after a series of bad decisions that put them in a corner with poor options. There probably really should be 3 or more sensors.
Without any ability to differentiate bad from good input, you may be better off simply relying on a single sensor at any one time and completely ignoring the other, at least as far as controlling MCAS is concerned. Tossing up a warning that says the sensors disagree is another matter.
I agree that one can argue that this doubles the chance of system shutdown compared to properly detecting that one sensor fails. That's obviously only ok if the system is expendable to a certain degree.
The elevator feel system (EFS) uses multiple sensors, in fact it has its own pitot probes[1] (two of them) on the elevator. It's not clear if it uses both, but I suspect it also uses both alpha vanes up front like MCAS should be doing. Input from the alpha vane (AoA sensor) is new on the MAX, IIRC. His tweets were a bit ambiguous, but the elevator feel system does not change the trim at all (also EFS was non-op on the Lion Air bird).
It's almost certainly NOT a sensor problem. Look at the graphs from the black box on the Lion Air flight. The angle-of-attack was almost exactly offset by twenty degrees left to right[2]. That sensor was working just fine, but something else on the path was fucked up (bad wiring? bad ADIRU? both have happened before). The fact that the alpha vane was replaced and that didn't fix the angle-of-attack readings supports this. Also, the 737 NG and 737 MAX use identical alpha vanes. While the angle-of-attack data is less important on an NG, we haven't seen people come out of the woodwork talking about how common alpha vane failures are on an NG. I'm inclined to believe that they're not all that common.
The Lion Air flight crew did write up the sensor problems[3].
I saw another one of these "I'm a software engineer so I've got great insight" posts[4] and chuckled. It's a bit of swing and a miss as well. MCAS is hardly silent or subtle. The same trim wheels his Cessna have are present on the 737 and they're noisy[5]. MCAS doesn't quite override the pilot either, and you can see that on the black box graphs from the Lion Air flight[2]. Twenty times the pilot flying entered opposite trim to counteract MCAS and twenty times MCAS paused for five seconds. On the twenty-first time something changed and the plane crashed. It's not quite that simple and that's why there are active investigations.
1: https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAF...
2: https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...
3: https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt...
4: https://medium.com/@jpaulreed/the-737max-and-why-software-en...
You say that the MCAS is neither silent or subtle, but there is no light or master caution or anything, just the trim wheel turning and clicking. I’m not sure you’d notice it if you were wearing a headset and were being distracted by the stick shaker.
The reason they died on activation #21 and not #1 is that the effects are cumulative. Every time it activates it dials in another 2.5 degrees of nose down trim. Eventually you can’t pull on the yoke hard enough to hold altitude.
I agree that the jury is still out as to whether it’s a sensor issue per se or some sort of ADC or data transmission issue.
The stabilizer position was graphed on that preliminary report (see: pitch trim position). There was a slight trend towards nose down pitch, but no, at the time they lost control there's a sharp decrease in pitch that doesn't correlate with MCAS. If you look at the "trim manual" line you'll see that the pilot flying was continuing to input nose up pitch on the switches to no effect. Something changed and it wasn't simply that the pilot was overridden by MCAS. After 21 even MCAS continued to counter the pitch up inputs as the stabilizer continued to contribute additional nose down pitch.
I agree that the jury is still out as to whether it’s a sensor issue per se or some sort of ADC or data transmission issue.
To be clear I highly doubt the alpha vane was defective. It's a possibility, but look at that graph. The left "angle of attack indicated" mirrors the right one almost exactly except that there's a near constant twenty degree offset. If that's a sensor problem in two separate sensors I'll pull a Werner Herzog and eat my freaking shoe.
BTW the trim wheel has little stripes on it, moves quickly when being operated automatically, and is fairly noisy. It will get your attention. However, my suspicion is that pilots experienced on the NG will mistake the trim wheel doing counterintuitive things for normal STS operation (especially after takeoff). I don't know if the Cessna trim wheel(s) are any quieter or more subtle, but the Cessna certainly doesn't have all the band-aids that a 737 NG or MAX would.
To this point here's a post from a 737 NG pilot[1]:
As a long-time 737 driver I'll just chime-in a few points
[...]
(1) Just after takeoff there is a lot going on with trim, power, configuration changes, and as noted above, the darn speed trim is always moving that trim wheel in seemingly random directions to the point that experienced NG pilots would treat its movement as background noise and normal ops. Movement of the trim wheel in awkward amounts and directions would not immediately trigger a memory item response of disconnecting the servos. No way.
(2) The pilots could very reasonably not have noticed the stab trim movement. Movement of the stab trim on the 737 is indicated by very loud clacking as the wheel rotates. On the -200 it was almost shockingly loud. On the NG, much less so. HOWEVER, the 737 cockpit is NOISY. It's one reason I am happy to not be flying it any more. The ergonomics are ridiculous. Especially at high speeds at low altitudes. With the wind noise, they may not have heard the trim wheel moving. The only other way to know it was moving would be yoke feel and to actually look at the trim setting on the center pedestal, which requires looking down and away from the windows and the instruments in a 'leans'-inducing head move. On the 717, for example, Ms. Douglas chimes in with an audible "Stablizer Motion" warning. There is no such indication on the 737.
[...]
Finally, runaway stab trim is a very, very rare occurence up until now. We trained it about once every other year in the sim because it is so rare. And when we did it was obvious. The nose was getting steadily heavier or steadily lighter with continuous movement of the trim wheel. That is a VERY different scenario than what these pilots faced.
We also trained for jammed stabilizer, the remedy for which is overcoming it with force. The information they were faced with could very reasonably have been interpreted that way, too.
1: https://www.pprune.org/rumours-news/619272-ethiopian-airline...
you need at least three sensors. (although the "sensors disagree" option you can purchase is better than nothing)
Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input?
Yes, it will probably make the system safer, but your guarantees become a lot more fuzzy. This is not the field for fuzzy guarantees.
You need to crash a lot of planes to train that NN.
Use of software to make a plane airworthy when it isn't natively airworthy is a problem. Civilian aircraft should have a different standard than fighter aircraft: one that allows graceful degradation under failure.
Having a second angle of attack sensor on the Boeing 737 MAX (which is critical to aircraft safety) be an extra cost option is unconscionable.
See https://pdfs.semanticscholar.org/d4ac/17fcc0db3396a3219785bd...
It's a question of what design choices you make. For instance Boeing only has three in its comparable 777 fly by wire system.
I call BS on that line of thinking. At the very least it seems like an easy decision to require redundancy in the AoA vane before letting the planes fly again.
Another crash related to the 737 MAX crash, is Scandinavian 751 [1]. On a MD-81 airliner, with "Automatic Thrust Restoration", A system the pilots were not trained on, caused the thrust to be increased, when the pilot tried to throttle back the damaged engines and clear the compressor stall. "Pilot Betrayed" was the Air Crash Investigations episode.
[0] https://en.wikipedia.org/wiki/Air_Inter_Flight_148
[1] https://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_7...
Giving away the liability and then up selling the cure is immoral.
Most car manufacturers called these "safety packages". These optional packages were sold basically in the same way as you jokingly mentioned above. "Buy this optional package and when you hit a tree at 50mph, this will keep you from flying through the windshield and will probably save your life."
I actually remember a conversation that my parents had with eachother when I was young about whether it was "worth it" to buy airbags in their new car. Yes, that was a REAL conversation that Millions of people had and a decision they had to make for nearly 25 years before airbags became standard in cars in the late 90s.
Lots of people felt that the optional saftey package weren't worth the money and many people died through the '80s and '90s from car crashes that they would have survived had they splurged for the "safety package" when they purchased their car from the dealer.
It turns out that when people are buying cars, they aren't planning on crashing them. So spending extra money that only helps them when they crash felt unnecessary. These packages could cost in the neighborhood of $2,500 extra. Which on a $20,000 car is over 10% extra! Would you pay 10% extra for a safety package if cars today didn't come with standard safety equipment?
This is what led to the USA making airbags mandatory in 1998, forcing this safety feature on manufacturers in order to save lives.
I imagine the 737 AoA sensors were sold as the same thing. Boeing sold this as a "additional redundancy package" or "safety package" and probably included many other features as part of a larger package.
Just like thrifty car buyers of the 70s, 80s, and 90s who figured they didn't need airbags in their cars and opted out of the safety package. The budget airlines of Lion Air and Ethiopian Airlines had a good history with the normal 737 planes and figured this additional safety package wasn't worth the money.
That’s like saying that car manufacturers decided to add a light saying their ABS is disengaged depending on whether you paid for it.
That’s more like it.
As far as I'm aware, all Boeing 737 aircraft have 2 AoA sensors. There isn't an option to add a 3rd redundant one like the Twitter poster suggests.
From what I understand, what is optional is the display of the AoA on the main screen and/or HUD, and the "AOA DISAGREE" warning which cross-checks the data from the 2 sensors.
(Airbus A320s have 3 AoA sensors. A380, A350, and Bombardier C-Series, aka A220, have 4.)
Pretty sure I'm misinterpreting things (I hope!).
In the other hand, if there are two sensors, what the control system does when they disagree and there is no hardware to show this to the pilots?
The MCAS system, apparently, always takes its input from one of the two AoA sensors - it alternates each flight. Lots of details here: http://www.b737.org.uk/mcas.htm
That's a potential reason why an unaware pilot wouldn't disengage the auto trim system. Because he wouldn't think the trim system as being capable of causing an extreme pitch down.
Also this response.
> Interesting. So how did Boeing get a plane without redundancy in a critical sub system approved?
Answered own question, by hiding it in what is normally a non-critical subsystem.
Problem is, if you need to inform the human, you also need to train the human, which leads to extra costs.
Except they can't because all pilots know about those since day 1 of their career as opposed to the MCAS sensor-software system designed to deal with airframe engineering issues.
Not much different than how with most car companies, the only way to get headlights that don't blind oncoming cars is to pay to upgrade to leather seats. C.f.:
https://www.iihs.org/iihs/news/desktopnews/night-vision-head...
To extend (torture?) the automotive analogy, it would be as if the manufacturer substituted a new braking system that required preheating the rotors, installed software to automatically ride the brakes for the first 2 miles of driving in order to get that preheat done, then sold an optional safety feature to verify that the brakes were at the proper temperature.
The old way worked just fine without need for “emulation”. The new airframe required some software to mimic the old one, but they decided to charge you for the add-on to make sure that their band-aid worked reliably.
https://www.nytimes.com/interactive/2019/03/13/world/boeing-...
Look at the charts above. Both pilots "fought" with the controls, correcting the plane course, then believing "it's corrected now" and then the plane sunk again and again.
Yes, lane-keeping and auto-braking are not mandated. I expect they will be within a few years.
I think I could have avoided this situation by turning on the left-turn signal to disable the lane-assist and generally its a good idea to use the indicator even when just passing a bike -- but that's not something I normally need to do to avoid this particular failure mode ...
That's exactly what you had to do according to the regulations.
That thing has a good default that is still overridable by the brute force (force that's less than the force needed to steer if the power assist fails altogether).
The adaptive cruise control in my VW actually gets confused sometimes. Drove through a puddle at the beach last year and it jammed on the brakes. Glad nobody rear-ended me.
> Boeing sells an option package that includes an extra AoA vane, and an AoA disagree light
The fact that the redundancy of a sensor on which a system capable of sudden, large control inputs relies is an optional package to be purchased separately... I simply have no words.
How was this package advertised in the brochure? Pay extra and when the airplane nosedives at high speed, this useful indicator will helpfully warn you it's because AoA reading disagreement?