Yep, 3 sensors can fail too. Less often however.

These sensors exist to solve a problem with the MAX design. Changing and moving the engines increased the likelihood of a stall when the engines could push the nose up (as I understand it). Fine.

But here's the kicker: this should be something that pilots should be trained on. They should be aware of how the MAX is different to the previous 737s and know what to do to disable this system if it causes problems.

But that might be the end of a common type rating, which is something the airlines (and apparently Boeing) didn't want.

The whole 737 MAX situation just looks like a giant clusterf--- that was a kneejerk reaction to the unexpected success of the A320neo to which Boeing had no answer and to avoid years-long development delays and losing more customers to Airbus, it really looks like they made shortcuts. And that's so damaging to their brand it really defies belief.

I am puzzled by the apparent lack of a "this plane is heading straight into the ground" sensor that should have overridden the MCAS.

The MCAS is designed to prevent an aerodynamic stall; a situation where the plane is pointed too far up and as a result starts heading straight for the ground. It does do by detecting the plane pointing too far up, and pushing it back down. If it did not function when the plane was descending rapidly it would not be fit for its designed purpose.

> The MCAS is designed to prevent an aerodynamic stall [...]

This has become a trope and is flatly incorrect.

MCAS was cooked up to maintain the handling characteristics required by the certification specifications.

The standard requires stick resistance to increase as the critical (stall) angle of attack is approached. The MAX violated that requirement due to the aerodynamics of the new engine cowls, so MCAS was Boeing's (?crap) implementation of "artificial feel" to meet the spec.

Yes, while the spec exists presumably to prevent the pilots from themselves taking the ship into a stall, MCAS doesn't change wing or aircraft performance in a material way that would be meant by "prevent a stall."

There seem to be a handful of people spreading this talking point here. In every authoritative source, MCAS is described as anti-stall system.

I believe the critical passage is here in CFR 14 §25.175¹

(2) With the landing gear retracted at low speed, the stick force curve must have a stable slope at all speeds within a range which is the greater of 15 percent of the trim speed plus the resulting free return speed range, or 50 knots plus the resulting free return speed range, above and below the trim speed (except that the speed range need not include speeds less than 1.3 VSR1, nor speeds greater than the minimum speed of the applicable speed range prescribed in paragraph (b)(1), nor speeds that require a stick force of more than 50 pounds)[...]

The crucial phrase is "stick force curve", which is the characteristic MCAS was created to tweak.

As far as 'authoritative sources' go, I'll just mention the Gell-Mann amnesia effect.²

(me: non-active aircraft dispatcher)

¹—https://www.ecfr.gov/cgi-bin/text-idx?node=14:1.0.1.3.11#se1... ²—https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect?wprov...

> The crucial phrase is "stick force curve", which is the characteristic MCAS was created to tweak.

How do you know this? Haven't seen that reported anywhere but in user comments here.

Define authoritative source.

When speaking to the uninitiated, anti-stall system gets the point across.

When speaking reguspeak, and actual aerodynamics, MCAS is a compensatory system that is meant to allow the plane to be certified airworthy. It is the same class of system as a mechanical stick-pusher (though even less effective since all it does is changes the yoke pressure required to bring the plane to a stall).

It's not a meme sadly, which makes this all the more tragic in hindsight. Sibling poster has quoted the CFR.

> MCAS is a compensatory system that is meant to allow the plane to be certified airworthy

Where did you get this information? My understanding is the airframe has the potential to nose up due to extra lift provided by engine placement (eg, lift not commanded by stick pull). MCAS is used to prevent this stall condition during high AoA.

From an understanding of control systems.

One of the most important things to get right with a control system is to maintain some level of consistency throughout the entire range of the system you're actuating.

Get on your computer, and move your mouse around. Now go into the settings and muck with your mouse sensitivity/acceleration curves, and try to drop your cursor on a particular file.

Odds are, you'll have some degree of difficulty until your brain adjusts to the new settings.

Same thing is relevant with airplane yokes. The force required to deflect a control surface (and therefore the plane) in an airstream is a result of the forces imparted on the surface + plane while by the impinged airstream.

Now, a force response curve should be smooth and predictable. Some distance of deflection on the yoke at one position should yield approximately the same level of deflection from the yoke at another position. If there is a sudden change in the number of degrees of AoA you get per degree of deflection, this is a highly undesirable flight characteristic.

For a pilot's perspective, I recommend the D.P. Davies Interview Podcast from the Royal Aeronautics Society, specifically the Boeing episode where he describes his certification flights of the Boeing 727-300.

That plane had a similar interaction with high lift devices at high AoA, making the plane more eager to pitch into a stall when you got to a certain point AoA-wise. This was in direct violation of this CFR. As Davies puts it, "You can't certify a plane that wants to stall itself!"

The 727 was eventually certified though when they added a mechanical stick pusher which would shove the stick forward when the pilot was getting close to that flight regime to make sure they didn't stall.

MCAS, by all indications, serves a similar purpose, it detects current AoA and trims the plane to counter the extra lift so that the deflection of the elevator at the top of the AoA curve has to fight the nose-down trim, which to a pilot would feel like the plane takes the same amount of force to deflect for those last few degrees. Basically, to use the mouse analogy, it's bumping down the sensitivity of the yoke to compensate for the extra sensitivity created by the lift at high angles of attack.

All of this is fine and dandy at 17 or so degrees AoA. Not so much though in attempted level flight with a busted AoA sensor.

Much of the art behind control systems comes from translating technological activity into humanly processable control schemes. That CFR is a common sense guideline basically specifying that a transport plane must have a predictable response curve.

It reminds me of the way nuclear reactors work, which kinda made it click. There they make criticality dependent on neutrons donated by fission byproduct decay, allowing for control rod situations, and thereby criticality level changes to take place over minutes instead of seconds.

Plus, look at the name

Maneuvering Characteristics Augmentation System.

The important takeaway is it doesn't really do squat to make stalls not happen beyond making it increasingly impossible for the pilot to get enough oomph out of the control surfaces to get the plane in stall and stay there. I don't know if there is also a stick pusher for MAX aircraft to help recover from a stall.

Given that the MCAS operates in intervals and does not provide constant input, it doesn't really stand that it's similar to a stick pusher as far as maintaining constant response. It seems as though it will make an evaluation and push the nose down regardless of how much stick input there is.

If the MCAS was constantly variable, like a mechanical stick pusher, then what you're saying would make sense. IMO, as reported, it seems like an 'anti-stall device' that automatically adjusts the plane's pitch regardless of stick input. Depending on conditions, this presumably could be little to no stick input.

That's what is so insane about this system. It was put in to make the thing act like a 737 within a very small part of the flight envelope. It never should have kicked in in normal flight.

But AoA sensors freezing is apparently common enough where this system is frequently kicking in when the plane isn't in the situation it was designed for.

The pilots were trained on how to deal with a runaway trim stabiliser. The procedure hasn't changed from the old 737, the only thing that has changed is that it is that the failure mode is more likely to occurr on the 737 MAX.

From the article: “A properly trained pilot should be able to solve an MCAS anomaly or any uncommanded flight-control input through procedures that are taught to all 737 pilots,” said Menza, noting that the emergency information Boeing distributed in December reiterated those procedures.

Here's a video of a competent 737 pilot showing those exact procedures: https://www.youtube.com/watch?v=xixM_cwSLcQ

Obviously the MCAS system is badly designed, but any automation system can fail and pilots need to know how to deal with it.

Per initial reports the black box data suggests that the procedure was followed, but the MCAS was re-engaged, looks like investigations are ongoing if the MCAS can re-engage automatically or if the trim was too hard to manually override and re-engaged by the pilots to be able to use the electric trim to level the plane.

https://www.wsj.com/articles/ethiopian-airlines-pilots-initi...

That actually raised more questions than it answers.

Did the pilots fail to notice the MCAS problem before it put the plane too far out of trim?

Could they have saved the plane if they stuck to the procedure and kept adjusting the trim by hand?

Why does MCAS override even direct pilot trim up commands?

Is the runway trim procedure even viable on older 737s once the trim goes past a certain point? Or are we just lucky that it never happens?

It doesn't present as runaway trim, though. It's a small change in the trim, repeated every few seconds, which can be counteracted on the control column but will eventually add up. Small trim changes are usually happening all the time.

This is my understanding.

In a typical run-away trim, the trim wheel will move a great distance. It's very noticeable, both audibly (the wheel makes a clack-clack-clack noise) and visibly (there are white paint flashes on the wheel). And, obviously, the plane nose goes up or down by more than expected. The correct remedy is to disable the auto trim control via switches on the panel (located near the trim wheels).

The MCAS will adjust the trim in small increments every 10 (or is it 20?) seconds. Yes, a pilot should notice this, but because it's intermittent, it's more likely they don't "see" it as run-away trim, and just a slightly abnormal trim (EDIT - problem made worse because pilots were not informed MCAS existed - it's not a failure more they have trained on). They may attempt to remedy this with the manual trim control (a rocker switch on the control yoke). This does NOT disable MCAS, it only re-trim the plane. MCAS will re-engage due to faulty AoA sensors repeatedly until either the pilot disables all auto-trim with the switch on the panel OR the plane runs into the ground.

Yeah. I'm also not clear on the practicality of retrimming the aircraft at low altitude (Ethiopian was never >1000ft above ground) before impacting terrain after you disable the trim motor. You have to use a hand crank, because you just killed the trim motor altogether, and I think it takes several minutes (!) of cranking to get the jackscrew from one end to the other.

I wonder if they killed the trim motor and then failed to reach level flight before the ground got in the way.

Fatigue as well. Looking at the flight track after the last and final nose down, it's they were overcome. Nose down -> more speed -> more stabilizer nose down force -> more nose down attitude even without a change in stabilizer -> more speed -> more stabilizer force -> more nose down attitude.

And then the ground got in the way.

I was hoping to have the damage in the Boing brand reflected more significantly in the share prices.

Maybe the problem hasn't been shouted from a high enough rooftop yet.

Or more likely it doesn't really matter, and in a couple years nobody will really remember this about Boeing. They'll go back to being a big airplane producer that has a stunning safety record, which they are despite this issue.

The airline industry is growing and there are only two manufacturers who can build hundreds of planes on time. Airbus is already near full capacity. It would be impossible NOT to buy Boeing if you want planes.

Which is also the correct thing to do. Despite our president and some NYTimes columnists being scared of automation (apparently they only agree when they're wrong), automation has saved so many lives that a failure like this doesn't affect the big picture.

Yes, mistakes were made, yes, we should find ways to avoid this happening in the future, no, automation isn't the enemy. Without automation in flying at the current number of airplanes in the sky we'd probably have one or two crashes a day. The fact that 2 crashes in half a year make the news is thanks to the amazing safety of modern airplanes.

Automation is definitely not the enemy.

However, automation that is designed in a criminally negligent manner is. People who create such designs should be put in prison, and we should not be using products from them ever again.

Have fun using any product. If your company gets big enough the probability that one of its products was faulty because of negligence slowly approaches 100%.

Any big company has a scandal, learns from it, the people that learned the lesson leave at some point, a new generation of employees makes a similar mistake again, wash rinse, repeat.

You probably can't fly any airplanes if you will not use any airplanes from a company that had a scandal like this.

AFAIK the automation had no good reason to be there first place, unless you consider racing Airbus for contracts a good reason. I mean, the simpler the better especially when lives are at stake, right?

If there were specific steps that the pilots could have taken to avert disaster, why not just incorporate them into the flight software? Why resorting instead to "reprogramming" the pilot? That's what training is: programming of the human brain.

One can't teach others without having first learned the lesson oneself. I'm not convinced Boeing has. I'm not convinced the company had adequately consider all possible scenarios and how pilots would react in them based on their intuition.

Even if it has enough sensprs, the flight software has no situational awareness.

This is by design. The flight software is deliberately kept simple enough that the pilots can be trained to understand the full workings of the flight software and what it will do in any situation.

The entire flight control algorithm is probably only a few hundred lines of psudocode.

It's a catch 22:

We are fully capable of designing a plane that safely flys itself from airport to airport without any pilots, to handle most emergency situations. We have been able to do so since the 80s.

But we are not able to put a pilot in the loop on such a plane without massive safety implications. To have a pilot in the loop, the software has to be kept super simple so the pilot can diagnose it in an emergency and take the correct action.

Either we have the flight software that is situational aware and authorised to take any action, or we have a super simple software and a pilot in the loop. There is no safe middle ground.

Not sure about the "able to do so since the 80s" part, but generally, I agree.

It's the same problem we see with Tesla, Uber, etc and autonomous cars. If it's to be truly autonomous, it needs to do so without ANY human intervention. As soon as a human is added to the loop, the system needs to be understandable by any human operator.

In the case of the MAX, even with Redundancy sensors, there is the possibility enough of them fail that the plane doesn't "know" how to resolve the problem. So, control must be returned to the pilot with enough time and enough information to avert disaster.

The USSR were flying their Buran space shuttle in the 80s with full autonomous capabilities.

It could launch into space, orbit, de-orbit, fly though re-entry and land at a runway without any external control (it was designed to complete it's mission under signal jamming conditions).

It had full situational awareness and could autonomously make decisions like redirecting to a different runway or airport depending on weather and other factors.

Maybe it's a bit of a stretch to say Buran was safe enough to fly passengers autonomously, but it was in the right direction.

Compared to a self driving car where you have to detect respond to other cars and solid objects, the amount of processing power to maintain a dynamic aerodynamic flight model and fly under instrument conditions is actually pretty low.

If all three AoA sensors are mounted to the side, they're likely to freeze at the same pitch if they do freeze. So unless you have multiple different ways to measuring AoA three sensors do not protect against this failure mode better than two does.

> These sensors exist to solve a problem with the MAX design.

No, the sensors existed before the MAX they are used by the autopilot system. With the MAX they are now also used by the MCAS system.

Triple-redundancy isn't 100% foolproof, but it is a lot better than two potentially-faulty sensors.

I know this isn't a great example, but triple redundancy is enough of a design cornerstone that Arthur C. Clarke made it one of the defining features of Rendezvous With Rama, a classic sci-fi story about an Oumuamua-like object that people manage to land on:

>It was now strikingly apparent that the "city", like so much of Rama, was triplicated. It consisted of three identical, circular complexes or superstructures, rising from a long, oval foundation. Photographs taken from the hub had also indicated that each complex was itself divided into three equal components, like a pie sliced into 120-degree portions. This would greatly simplify the task of exploration; presumably they had to examine only one-ninth of [the city] to see the whole of it.

It doesn't exactly sound like a new or controversial idea, you know?

The problem with this idea is the "identical" aspect of each redundancy. If it's identical, then it will likely suffer from the same failure mode at the same time as the others. In the case of the AoA freezing, that's exactly what happened.

A more robust idea is to have three redundant subsystems, each different.

Yeah, but I get the feeling that it's more of a way to ensure that you probably get at least two components that will 'burn in' well and hopefully not fail too early:

https://en.wikipedia.org/wiki/Bathtub_curve

Like, you don't expect them to last forever, but you do expect at least 2/3 to meet their lifetime estimates.

I once had a discussion with a scifi author about how to design a starship to last a long time. I suggested that it being repairable would be of paramount importance. To that end, the various systems should use interchangeable parts. Starting with the obvious like nuts&bolts should be standardized, extend that to things like computer boards and components, motors, etc.

For example, if you had a linux machine running the fusion drive, and a mac machine running the entertainment arcade, and the linux machine shorted out, it would sure suck if you couldn't cannibalize the entertainment arcade computer to fix it.

It goes back to Apollo 13, where the CO2 scrubbers on the command module were a different design than the ones on the lunar module, which inhibited them being able to refresh the ones on the lem, and nearly caused loss of the crew.

It's the same for software. Software that's supposed to be used "forever" should be written such that parts of it can be easily replaced. Highly flexible designs often lose against simpler designs because it's easier to rewrite the latter and very hard to change something when you bump against the limits of flexibility in the former.

I had a similar discussion once, and the result was very biology cells inspired. Any subsystem must be able to repair itself, and must state its defunct state on repair needed state. If that times out the subsystem must destroy itself "deactivation". Dependant on importance other subsystems must pick up the load and create a replacement under strain over time.

In a plane, the aoa could be replaced by sensor fusion of gyroscopic sensors and air density sensors distributed over the hull.

Selfrepairing could be a achieved via a reservoir of peeble like standardized spare parts traveling through a conduit and applied via vibration and standardized interfaces. Final fallback, a simulation of expected circumstances, suggesting a half way accurate fallback model.

The difference with a plane, however, is it isn't necessary to fix it. It's just necessary to get it on the ground safely.

I don't think it's so much about the lifetime of the component, or whether it eventually fails due to faulty manufacturing or gradual wear and tear. (What the bathtub curve gets at.)

It's that if you do make parts that are identical, and you expose them to identical unanticipated circumstances, they're likely to behave identically, and fail identically.

I've seen this w/ distributed databases. I overloaded a Cassandra instance once, causing it to OOM. That node failed, but it's distributed, so another node automatically took over, ran the same query, and OOMed, and then the last guy took the query, and very quickly I had 3 dead nodes.

This is especially true with software. Having identical software in the backup means it'll seg fault too.

That's science fiction, not science. Clarke used three to push your brain past our sense of left/right symmetry not because of some idea related to redundancy.

I've had to do a bit of functional safety stuff recently and one of the things they focus on are "common cause failures" (of which this is a perfect example - if one of the sensors freezes, the others are likely to do so as well). If you want to be sure something works, it's not enough to just have two or three copies of the same system. You have to have redundancy in manner of operation as well.

To be specific, I'd call that functional redundancy, versus hardware redundancy. Be able to get the same thing done in different ways.

NASA's Genesis mission was bit by this a while back. It had redundant parachute sequencers for a sample return capsule, but they were all assembled identically and with the G-sensors backwards. Consequently, they all failed in exactly the same way.

Couldn't one detect the frozen ones by simply noticing the loss of noise in the readings? I imagine an AoA sensor functioning normally to have a small amount of variation between readings due to turbulent air etc.

Alternatively, couldn't one correlate the AoA sensor output with the accelerometer output (or some other pitch sensor)? A frozen AoA would have identical readings as the plane changes pitch, a functioning one shouldn't, no?

Although a stuck AoA sensor is also going to have a very characteristic signature - no change in reported AoA. A working AoA sensor will always have some noise in the signal as the air buffets the plane. Similarly, if you maneuver the plane and the indicated AoA doesn't change at all, you can deduce the sensor is stuck and reject it (or turn on the heater, or maybe have a mechanical way to force the AoA sensor through its range of motion, etc).

If there are three of the same type (make/model?) of sensor teaming to vote on the status being measured I'm not suprised that they can suffer the same failure (iceing) at the same time causing an incorrect state to be upvoted.

I would have thought that for any condition there would be more than one mechanism available to sense it. You could get different devices to vote on what is the real condition.

This has been done for years in visual cockpits with regards to an artifical horizon. The pilot only has to look out the window (assuming visibliity) to confirm what the instrument is reporting. Of course when the pilot loses visibility and starts to rely on their inner ear for "level" bad things can happen.

Many more simple planes, especially things like gliders, have a simple ball bearing in a curved tube to indicate "down" as well as the more convential artifical horizon.

Are you talking about a turn coordinator? That doesn't show "down." The sum of forces acting in a coordinated turn will keep the ball centered even though the airplane is banked.

Also I'm not sure what you're talking about with using the real horizon as a backup to the artificial horizon. It's not meant to be a redundant setup. Artificial horizons aren't required for visual (looking outside) flight, and relying primarily on the instruments in visual flight is an error.

In clouds/poor visibility, the cross-check for the artificial horizon is provided by the other flight instruments: airspeed, altimeter, turn coordinator, vertical speed indicator, and directional gyro (or their equivalents on an electronic display).

The point is that you are using a different type of device to cross check the reading from the first. In the case of the faulty sensor, it sounds like they are using the same type of device therefore conditions that impact the effectiveness of that device will likely impact more than one of them.

> Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch.

If it's typical for the sensor's data to vary from 30.00 to 30.99 degrees as the wind passes over the wings and two sensors are permanently frozen at 10.42, after 8 minutes I think that there might be a high enough probability for the computer's software to decide that two sensors are broken and one is working.

I'm admittedly ignorant of aircraft design but years ago when I first found out what those little AoA fins do, I was like, why the heck aren't there more of them plastered all over the place? Like, more than 3.

It does seem like a primitive system to use the wind for AoA, seems like there should be better and more reliable ways to determine it.

Update: AoA not AoT

That's what AoA is though. It's the relative wind angle over the wing. The wind matters. You can't calculate AoA simply from pitch or vertical speed or something like that.

IIRC Boeing calculates a synthetic AoA value from the inertial reference unit in the 777 and 787, so you can derive it.

Is there another method to measure wind direction, that doesn't involve a small physical winglet?

Another way would be to stick a tab in the slipstream, and measure the pressure on its leading edge and both sides. Or just put a strain gauge on the mount of the tab and measure the bending forces (and this would have no ports to freeze or moving parts to jam). I bet there are lots of ways.

P.S. I bet you could even put a strain gauge on the wing spar to measure the bending force on the wing, couple that with other inputs like airspeed, and come up with a reasonable AoA value.

The problem with such approaches is that you have to model all the environmental and performance factors that could cause an identical pseudo-AoA to be calculated, in all flight modes. And to do that you need a range of sensors, all redundant... so now you need additional input validation and sanitisation.

For example, measuring slipstream over the wing is not sufficient. You need to known the pressure ratio between above and below, so there's two sensors. But airflow over a wing is not contiguous or straight, so you need more spanwise sensors to start averaging. And the inboard wing will stall first and give erroneous data, so you need to detect that and disregard it...

Rather than just measuring the raw value at source.

No need to measure pressure at the top and the bottom if you use a strain gauge on a wing spar to measure the net force bending the wing.

I'm sure a function can be constructed that takes that input, along with the airspeed, and produces AoA. It can be used as a crosscheck with the AoA sensors. Since it takes airspeed as another parameter, it can also crosscheck the pitot tubes.

It can be derived using static ports that measure air pressure at several locations.

They need triple temperature sensors on those sensors to sense when they freeze...

Although you're being facetious, AoA sensors do have self-diagnostics that determine when their heating elements have failed.

Incodentally on the A380 and A350 Airbus have changed to combined AoA-pitot sensors of which there are now four.

Sounds like Airbus airplanes are much, much safer to fly on than Boeings. Why would anyone trust a company which thinks it's OK to have a flight control system that only uses input from a single sensor that's known to fail?

you're hired!

Ah, it's turtles all the way down.

They should have used (n+1)-th prime sensors, dammit. When will they ever learn?

a customer acceptance fight is a shitty way to go