News for hackers. Stuff that matters.
Re:737 Max Explanation by a Software Engineer by JorgeGT on Sunday March 17, 2019 @01:45PM (#19415219)
That... makes it even worse? Why would someone on Earth disregard an available extra sensor for a system that has unlimited authority to bring the horizontal stabilizer to full deflection?
Re:737 Max Explanation by a Software Engineer by ams6110 on Sunday March 17, 2019 @01:56PM (#19415292)
Yeah I don't disagree. I don't fly, but have sort of been following this out of curiosity. One comment I've seen from some pilots is basically "ehh, runaway trim isn't a new thing, we train for it in the sim, and there's a standard way to deal with it (disengage the automatic system and trim manually)."

So perhaps Boeing felt that this didn't really change anything in that regard. They seem to have been proven wrong.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Sunday March 17, 2019 @03:42PM (#19416050)
Technically MCAS activation isn't runaway stabilizer trim, and I'm sure that tripped up the Lion Air pilots. Check out the quick reference for a runaway stabilizer[1]. Note the steps:

* Control airplane pitch attitude manually with control column and main electric trim as needed

* If the runaway stops, stop.

Well, electric trim input on the yoke will stop MCAS temporarily. Of course if these guys had any experience on an NG they're already used to the computers trimming the plane in a counterintuitive manner via the speed trim system (STS). So not only is MCAS not a runaway trim situation, but pilots flying the NG will get used to the computer trimming the stabilizer "at random".

1: http://www.737ng.co.uk/737-800%20Quick%20Reference%20Handboo...

Re:737 Max Explanation by a Software Engineer by mannykannot on Sunday March 17, 2019 @04:15PM (#19416279)
This sort of MCAS failure presents itself to the crew as a trim runaway, and the instructions in the document continue:

  4 If the runaway continues:

      STAB TRIM CUTOUT switches (both)   . . CUTOUT

      If the runaway continues:

        Stabilizer trim wheel    . . . . .   Grasp and hold

  - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  5  Stabilizer    . . . . . . . . . . . .   Trim manually
(That bit about grasping and holding the trim wheel was a surprise to me.)

This is what the Lion Air crew of the flight before the one that crashed did, in response to the same sort of MCAS failure, and completed the flight without further trouble. While Boeing made a serious error in hiding the differences between this variant and its predecessors, it seems that the prior trim runaway procedure does work for this sort of MCAS failure (unless evidence to the contrary comes out of the Ethiopian Airways investigation.)

Re:737 Max Explanation by a Software Engineer by erichocean on Sunday March 17, 2019 @06:48PM (#19417257)
Boeing is "right" in the sense that, if you follow the normal 737 flight manual, you'll be fine, even with MCAS enable or even MCAS misbehaving due to the sensor readings being bad or whatever.

I think the problem is that the symptoms are different from other 737s, and if you don't KNOW about the MCAS stuff, that's very surprising. So as humans, even though logically the same procedure applies, instinctively you don't think to do it. And given the very short time frames involved, and the margin of error, it's not too surprising that these events have occurred.

Perhaps the need-to-retrain criteria needs to include the following amendment:

If the craft exhibits new symptoms, even if the response to those symptoms are covered correctly be the old operations manual, pilots will need to be retrained to account for the new symptoms.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Monday March 18, 2019 @12:08AM (#19418626)
This sort of MCAS failure presents itself to the crew as a trim runaway

No, it doesn't and that's the problem. A runaway won't stop when you hit the trim buttons.

Re:737 Max Explanation by a Software Engineer by WalterBright on Monday March 18, 2019 @01:39AM (#19418917)
> A runaway won't stop when you hit the trim buttons.

Depends on what causes the runaway. If the autopilot goes berserk, the trim buttons will override it (if I recall correctly, it's been a long time).

Re:737 Max Explanation by a Software Engineer by salawat on Sunday March 17, 2019 @03:34PM (#19415987)
No simulator accounted for a borked AoA sensor being able to nosedive the plane, or the altered aerodynamics from the different engine configuration.

So while yes, runaway trim is a thing, the circumstances under which an error could happen are substantially different.

For instance, a pilot could check the maintenance log book and see some work or inspection was done on the auto-trim subsystem. This would prime the pilot to be more on the alert for the possibility of trim misbehavior during the flight. Crisis averted, right?

But no pilot would look at an entry for an AoA sensor being off or worked on, and would think, "Boy, better look out for that safety system I never knew existed that could cause a trim runaway because my previously airworthyness agnostic AoA sensor is unreliable."

Flying and complex system diagnosis in time sensitive conditions requires extensive ahead of time mental model creation ahead of time.

Re:737 Max Explanation by a Software Engineer by lutorm on Sunday March 17, 2019 @03:52PM (#19416128)
It is weird, though, that this was not caught in simulations (not training simulations, but in design simulations.) I would assume that running simulations to see how the design behaves when sensors fail would be an absolutely standard thing to do. I don't work on airplanes, but we routinely simulate sensor failures and their effect on software systems and you'd think this would not be optional for passenger aircraft certification.
Re:737 Max Explanation by a Software Engineer by salawat on Sunday March 17, 2019 @06:00PM (#19416948)
When all of your testing is in house, and you don't have people who are willing to put the project at jeopardy for the sake of doing it right, you'll be amazed what can get overlooked.

If anything, the process of engineering is most difficult in that answering most questions tends to be straightforward (not easy, but straightforward once you know the right methods to apply), while figuring out if you've asked all the right questions is the thing that keeps me awake at night. If you don't ask and dig in until you've answered it fully, it's easy to get blind sided.

I have difficulty believing it could have gone so horribly wrong now, but I can't deny that just from the information available, even if MCAS isn't the root cause of the Ethiopian Air crash, there is some egregious failures in sound practice going on with Boeing to have slipped up this badly.

That's the thing with sound Engineering, when you do it, it just verks. When you don't...

Re:737 Max Explanation by a Software Engineer by CaptainZapp on Sunday March 17, 2019 @02:26PM (#19415462)
Key quote is training on a simulator

That's a tall order when the manufacturer doesn't inform their customers out of commercial expediency.

Re:737 Max Explanation by a Software Engineer by wahern on Monday March 18, 2019 @02:38AM (#19419094)
If you only have two sensors and you're taking input from both sensors, assuming each sensor has the same probability of malfunction then you're doubling the chance of processing bad input.

You could choose to select the input that results in the least-worst bad outcome. But that may not bring the total risk below the increased risk of utilizing both sensors. The sensors are doing something purposeful, after all.

So it may be logical what they're doing. OTOH, the logic may make sense only in a path dependent sort of way, after a series of bad decisions that put them in a corner with poor options. There probably really should be 3 or more sensors.

Re:737 Max Explanation by a Software Engineer by chopin on Monday March 18, 2019 @05:44AM (#19419747)
You take input from both sensors only when they don't disagree (by a margin). There may be failure modes where both have the same bad input but I don't think that this doubles your chances of processing bad input.
Re:737 Max Explanation by a Software Engineer by wahern on Monday March 18, 2019 @01:24PM (#19423048)
You have double the chance that one of the sensors is bad. Which one do you choose? The one that says AoA is normal? Maybe that's the one that is bad. Worse, maybe such a bad input is more likely in a stall scenario.

Without any ability to differentiate bad from good input, you may be better off simply relying on a single sensor at any one time and completely ignoring the other, at least as far as controlling MCAS is concerned. Tossing up a warning that says the sensors disagree is another matter.

Re:737 Max Explanation by a Software Engineer by chopin on Wednesday March 20, 2019 @08:28AM (#19441359)
I meant disabling the system if they disagree.

I agree that one can argue that this doubles the chance of system shutdown compared to properly detecting that one sensor fails. That's obviously only ok if the system is expendable to a certain degree.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Sunday March 17, 2019 @03:37PM (#19416014)
Because there was no available extra sensor. Trevor gets a few other things wrong too.

The elevator feel system (EFS) uses multiple sensors, in fact it has its own pitot probes[1] (two of them) on the elevator. It's not clear if it uses both, but I suspect it also uses both alpha vanes up front like MCAS should be doing. Input from the alpha vane (AoA sensor) is new on the MAX, IIRC. His tweets were a bit ambiguous, but the elevator feel system does not change the trim at all (also EFS was non-op on the Lion Air bird).

It's almost certainly NOT a sensor problem. Look at the graphs from the black box on the Lion Air flight. The angle-of-attack was almost exactly offset by twenty degrees left to right[2]. That sensor was working just fine, but something else on the path was fucked up (bad wiring? bad ADIRU? both have happened before). The fact that the alpha vane was replaced and that didn't fix the angle-of-attack readings supports this. Also, the 737 NG and 737 MAX use identical alpha vanes. While the angle-of-attack data is less important on an NG, we haven't seen people come out of the woodwork talking about how common alpha vane failures are on an NG. I'm inclined to believe that they're not all that common.

The Lion Air flight crew did write up the sensor problems[3].

I saw another one of these "I'm a software engineer so I've got great insight" posts[4] and chuckled. It's a bit of swing and a miss as well. MCAS is hardly silent or subtle. The same trim wheels his Cessna have are present on the 737 and they're noisy[5]. MCAS doesn't quite override the pilot either, and you can see that on the black box graphs from the Lion Air flight[2]. Twenty times the pilot flying entered opposite trim to counteract MCAS and twenty times MCAS paused for five seconds. On the twenty-first time something changed and the plane crashed. It's not quite that simple and that's why there are active investigations.

1: https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAF...

2: https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

3: https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt...

4: https://medium.com/@jpaulreed/the-737max-and-why-software-en...

5: https://www.youtube.com/watch?v=3pPRuFHR1co

Re:737 Max Explanation by a Software Engineer by cameldrv on Sunday March 17, 2019 @11:54PM (#19418585)
I’m the OP Dave BTW. Trevor copied a Facebook post I made a couple of days ago into those tweets. I don’t know all of the details about the EFS system but I’d be pretty sure it doesn’t use data from both sides for anything, because that would be against the brick wall isolation concept in the air data and flight control computers.

You say that the MCAS is neither silent or subtle, but there is no light or master caution or anything, just the trim wheel turning and clicking. I’m not sure you’d notice it if you were wearing a headset and were being distracted by the stick shaker.

The reason they died on activation #21 and not #1 is that the effects are cumulative. Every time it activates it dials in another 2.5 degrees of nose down trim. Eventually you can’t pull on the yoke hard enough to hold altitude.

I agree that the jury is still out as to whether it’s a sensor issue per se or some sort of ADC or data transmission issue.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Monday March 18, 2019 @12:15AM (#19418657)
The reason they died on activation #21 and not #1 is that the effects are cumulative.

The stabilizer position was graphed on that preliminary report (see: pitch trim position). There was a slight trend towards nose down pitch, but no, at the time they lost control there's a sharp decrease in pitch that doesn't correlate with MCAS. If you look at the "trim manual" line you'll see that the pilot flying was continuing to input nose up pitch on the switches to no effect. Something changed and it wasn't simply that the pilot was overridden by MCAS. After 21 even MCAS continued to counter the pitch up inputs as the stabilizer continued to contribute additional nose down pitch.

I agree that the jury is still out as to whether it’s a sensor issue per se or some sort of ADC or data transmission issue.

To be clear I highly doubt the alpha vane was defective. It's a possibility, but look at that graph. The left "angle of attack indicated" mirrors the right one almost exactly except that there's a near constant twenty degree offset. If that's a sensor problem in two separate sensors I'll pull a Werner Herzog and eat my freaking shoe.

Re:737 Max Explanation by a Software Engineer by engire on Saturday March 23, 2019 @12:42PM (#19471164)
I totally agree here. Im an AME with Max Licence and i would have to say. Im quite sure this wasnt an AOA sensor. The FCC thats active with MCAS as its function (both have this Function) would have been looking at the right vane on one flight and then the left vane on the next flight, however on the last 3 legs the fault was the same. And trust me on this one. May have been different indications bbut all pointed at the same fault. I suspect the FCC or ADIRU was at fault or something it aircraft installation that was inducing the 20 degree difference between the 2.
Re:737 Max Explanation by a Software Engineer by autopilotsw on Monday March 25, 2019 @07:53AM (#19481741)
I don't believe there is a brick wall design here since the FCC outputs are not compared so why would you isolate inputs. I believe it's just a bad design to use the single AOA input.
Re:737 Max Explanation by a Software Engineer by inferiorhuman on Monday March 18, 2019 @10:56PM (#19427686)
You say that the MCAS is neither silent or subtle, but there is no light or master caution or anything, just the trim wheel turning and clicking. I’m not sure you’d notice it if you were wearing a headset and were being distracted by the stick shaker.

BTW the trim wheel has little stripes on it, moves quickly when being operated automatically, and is fairly noisy. It will get your attention. However, my suspicion is that pilots experienced on the NG will mistake the trim wheel doing counterintuitive things for normal STS operation (especially after takeoff). I don't know if the Cessna trim wheel(s) are any quieter or more subtle, but the Cessna certainly doesn't have all the band-aids that a 737 NG or MAX would.

To this point here's a post from a 737 NG pilot[1]:

As a long-time 737 driver I'll just chime-in a few points

[...]

(1) Just after takeoff there is a lot going on with trim, power, configuration changes, and as noted above, the darn speed trim is always moving that trim wheel in seemingly random directions to the point that experienced NG pilots would treat its movement as background noise and normal ops. Movement of the trim wheel in awkward amounts and directions would not immediately trigger a memory item response of disconnecting the servos. No way.

(2) The pilots could very reasonably not have noticed the stab trim movement. Movement of the stab trim on the 737 is indicated by very loud clacking as the wheel rotates. On the -200 it was almost shockingly loud. On the NG, much less so. HOWEVER, the 737 cockpit is NOISY. It's one reason I am happy to not be flying it any more. The ergonomics are ridiculous. Especially at high speeds at low altitudes. With the wind noise, they may not have heard the trim wheel moving. The only other way to know it was moving would be yoke feel and to actually look at the trim setting on the center pedestal, which requires looking down and away from the windows and the instruments in a 'leans'-inducing head move. On the 717, for example, Ms. Douglas chimes in with an audible "Stablizer Motion" warning. There is no such indication on the 737.

[...]

Finally, runaway stab trim is a very, very rare occurence up until now. We trained it about once every other year in the sim because it is so rare. And when we did it was obvious. The nose was getting steadily heavier or steadily lighter with continuous movement of the trim wheel. That is a VERY different scenario than what these pilots faced.

We also trained for jammed stabilizer, the remedy for which is overcoming it with force. The information they were faced with could very reasonably have been interpreted that way, too.

1: https://www.pprune.org/rumours-news/619272-ethiopian-airline...

Re:737 Max Explanation by a Software Engineer by bjowen on Sunday March 17, 2019 @10:28PM (#19418277)
One thing I’ve seen kicking around is that MCAS alternates input between the left FCC and right FCC, which might help understand why the JT43 only had to fight the stick shaker the whole flight (!) but JT610 had the automated trim input as well.
Re:737 Max Explanation by a Software Engineer by m463 on Tuesday March 19, 2019 @07:44PM (#19437379)
serious question: which sensor has failed?

you need at least three sensors. (although the "sensors disagree" option you can purchase is better than nothing)