News for hackers. Stuff that matters.
Re:737 Max Explanation by a Software Engineer by ams6110 on Sunday March 17, 2019 @01:38PM (#19415187)
This isn't exactly right. The aircraft has two sensors, but the system in question (MCAS) is only ever looking at one of them. That is part of what the software fix is reportedly going to address.
Re:737 Max Explanation by a Software Engineer by im_down_w_otp on Sunday March 17, 2019 @01:50PM (#19415256)
For a system with the responsibilities and impact on the safety-case that MCAS has, not even a 1oo2D design would be sufficient, since there's no clear way to determine which input is correct or incorrect if they diverge, but worse still is that MCAS doesn't seem to leverage even that basic, yet still insufficient, safety design from what I've been able to find on it.

At the end of the day the fail-safe is to throw control back to the human. And, then of course pin all the liability onto the human operator when things go wrong.

This was the problem I had with the Lion Air conclusion. Sure, it may be the case that a pilot can potentially override this system with situational awareness and training. However, the pilot didn't create this band-aid pile in the first place.

Re:737 Max Explanation by a Software Engineer by tzs on Sunday March 17, 2019 @02:43PM (#19415589)
> For a system with the responsibilities and impact on the safety-case that MCAS has, not even a 1oo2D design would be sufficient, since there's no clear way to determine which input is correct or incorrect if they diverge

Wouldn't that depend on how often MCAS is actually needed, and how it handles divergent sensors?

My understanding as a layman is that there are situations where a 737MAX is more likely to stall than a "regular" 737 would, but I haven't seen anything on how often those occur. Are they expected to routinely occur, so a fully functioning MCAS is important for safe operation of the plane?

Or are they supposed to be rare, one in several million events, and the MCAS is there to make handling those events the same as they would be handled in a "regular" 737, to minimize the need for retraining?

If the later, then I'd expect two sensors would be fine, or even just one sensor IF there was a reliable way to tell if it had failed, as long as the MCAS response to a sensor problem is to disable itself and let the pilots know they have to stay out of those "more likely to stall" situations for the rest of the flight.

Re:737 Max Explanation by a Software Engineer by salawat on Sunday March 17, 2019 @03:22PM (#19415920)
The problem comes in where the MCAS use should be infrequent, unless the sensor is borked.

In computing, we have a principle stated Garbage in, Garbage out. Circuits and electronics don't think. They compute. There is no error checking except that which is specifically designed and implemented into the system.

If you're getting data from a biased sensor with +/- 10 degrees AoA, a 10 degree actual AoA (well within the safe operating envelope) suddenly appears to MCAS as a 20 degree AoA (oh shit territory).

The system therefore engages doing exactly what it was designed to do.

Thus is the crux of the matter. The pilot was flying safely while his AoA sensor was telling a safety system he knew nothing about that he was flying dangerously.

The AoA system on earlier models of aircraft that the MAX was based on were a functional luxury/situational awareness aid. The flyability was not impacted by a horked sensor.

That changed once they had to add a software driven mechanism to keep the flight characteristics similar enough with the old airframe to be able to release the aircraft to 737 trained pilots, and not have to worry about retraining. Maintenance and pilot alike both needed to be aware that the AoA sensor became a safety critical component due to a failure or miscalibration jeopardizing the controllability of the airframe.

If they had gone through a full recert of this airframe, and not an expedited self-certify/grandfathering, these tragedies would have had a much smaller likelihood of occurring due to the increased scrutiny. Props on Brazil for doing their own footwork, and not blindly trusting the FAA's delegation to the manufacturer.

Personally, as a software engineer and quality assurance specialist, if I'd seen anything remotely like this come across my desk, I'd be raising hell, even if it meant yanking someone into a VP/C's office and giving them a dressing down for skimping on a cross-cutting safety critical concern, deadlines be damned.

Re:737 Max Explanation by a Software Engineer by Johnny555 on Monday March 18, 2019 @01:04AM (#19418796)
his AoA sensor was telling a safety system he knew nothing about that he was flying dangerously.

Is that true with the Ethiopian Airlines crash? I can believe it was true with Lion Air, but that crash was so widely publicized that I'm surprised that any passenger airline captain had not heard of the MCAS system.

Re:737 Max Explanation by a Software Engineer by chopin on Monday March 18, 2019 @05:39AM (#19419733)
You need to realize this problem before it is too late. If there is full trim down when you realize it there might not be enough room left to offload the stabilizer and trim back (manually, no less). Afaik if you cut off stab trim you can only trim back manually by moving the trim wheels.

I'd not be surprised if there will be another problem with MCAS detected with the ET flight where the pilots had effectively no chance despite knowing of the Lion air crash.

Re:737 Max Explanation by a Software Engineer by dilap on Sunday March 17, 2019 @04:01PM (#19416200)
yeah, I think you're right -- since you don't normally need the MCAS, you could probably get away w/ 2 AoA sensors and just disengage MCAS when they don't agree. you'd only get into trouble when you need MCAS and the AoA sensors are busted, which is a (rare) * (rare) event, so probably low enough odds to tolerate (i.e., probably never over lifetime of plane).

(just speculation of course.)

Re:737 Max Explanation by a Software Engineer by lutorm on Sunday March 17, 2019 @07:43PM (#19417543)
There are other ways of estimating the AoA. If you know the airspeed and the load factor, you can back out AoA to a reasonable accuracy. That's less precise than a dedicated sensor, but it should be good enough to tell which sensor is the good one if they diverge.
Re:737 Max Explanation by a Software Engineer by taneq on Sunday March 17, 2019 @08:07PM (#19417653)
> since there's no clear way to determine which input is correct or incorrect if they diverge

In systems with dual redundant channels you don't try to determine which one is correct (unless it's obvious, like lost comms to one sensor), you just throw an error and stop trusting both sensors.

Re:737 Max Explanation by a Software Engineer by im_down_w_otp on Sunday March 17, 2019 @09:56PM (#19418144)
It depends on whether the system needs to be Fail-Safe or fault-tolerant.

But, yes. Most of these systems are not designed to be resilient to that class of issues, and just send control back to the operator.

Re:737 Max Explanation by a Software Engineer by JorgeGT on Sunday March 17, 2019 @01:45PM (#19415219)
That... makes it even worse? Why would someone on Earth disregard an available extra sensor for a system that has unlimited authority to bring the horizontal stabilizer to full deflection?
Re:737 Max Explanation by a Software Engineer by ams6110 on Sunday March 17, 2019 @01:56PM (#19415292)
Yeah I don't disagree. I don't fly, but have sort of been following this out of curiosity. One comment I've seen from some pilots is basically "ehh, runaway trim isn't a new thing, we train for it in the sim, and there's a standard way to deal with it (disengage the automatic system and trim manually)."

So perhaps Boeing felt that this didn't really change anything in that regard. They seem to have been proven wrong.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Sunday March 17, 2019 @03:42PM (#19416050)
Technically MCAS activation isn't runaway stabilizer trim, and I'm sure that tripped up the Lion Air pilots. Check out the quick reference for a runaway stabilizer[1]. Note the steps:

* Control airplane pitch attitude manually with control column and main electric trim as needed

* If the runaway stops, stop.

Well, electric trim input on the yoke will stop MCAS temporarily. Of course if these guys had any experience on an NG they're already used to the computers trimming the plane in a counterintuitive manner via the speed trim system (STS). So not only is MCAS not a runaway trim situation, but pilots flying the NG will get used to the computer trimming the stabilizer "at random".

1: http://www.737ng.co.uk/737-800%20Quick%20Reference%20Handboo...

Re:737 Max Explanation by a Software Engineer by mannykannot on Sunday March 17, 2019 @04:15PM (#19416279)
This sort of MCAS failure presents itself to the crew as a trim runaway, and the instructions in the document continue:

  4 If the runaway continues:

      STAB TRIM CUTOUT switches (both)   . . CUTOUT

      If the runaway continues:

        Stabilizer trim wheel    . . . . .   Grasp and hold

  - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  5  Stabilizer    . . . . . . . . . . . .   Trim manually
(That bit about grasping and holding the trim wheel was a surprise to me.)

This is what the Lion Air crew of the flight before the one that crashed did, in response to the same sort of MCAS failure, and completed the flight without further trouble. While Boeing made a serious error in hiding the differences between this variant and its predecessors, it seems that the prior trim runaway procedure does work for this sort of MCAS failure (unless evidence to the contrary comes out of the Ethiopian Airways investigation.)

Re:737 Max Explanation by a Software Engineer by erichocean on Sunday March 17, 2019 @06:48PM (#19417257)
Boeing is "right" in the sense that, if you follow the normal 737 flight manual, you'll be fine, even with MCAS enable or even MCAS misbehaving due to the sensor readings being bad or whatever.

I think the problem is that the symptoms are different from other 737s, and if you don't KNOW about the MCAS stuff, that's very surprising. So as humans, even though logically the same procedure applies, instinctively you don't think to do it. And given the very short time frames involved, and the margin of error, it's not too surprising that these events have occurred.

Perhaps the need-to-retrain criteria needs to include the following amendment:

If the craft exhibits new symptoms, even if the response to those symptoms are covered correctly be the old operations manual, pilots will need to be retrained to account for the new symptoms.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Monday March 18, 2019 @12:08AM (#19418626)
This sort of MCAS failure presents itself to the crew as a trim runaway

No, it doesn't and that's the problem. A runaway won't stop when you hit the trim buttons.

Re:737 Max Explanation by a Software Engineer by WalterBright on Monday March 18, 2019 @01:39AM (#19418917)
> A runaway won't stop when you hit the trim buttons.

Depends on what causes the runaway. If the autopilot goes berserk, the trim buttons will override it (if I recall correctly, it's been a long time).

Re:737 Max Explanation by a Software Engineer by salawat on Sunday March 17, 2019 @03:34PM (#19415987)
No simulator accounted for a borked AoA sensor being able to nosedive the plane, or the altered aerodynamics from the different engine configuration.

So while yes, runaway trim is a thing, the circumstances under which an error could happen are substantially different.

For instance, a pilot could check the maintenance log book and see some work or inspection was done on the auto-trim subsystem. This would prime the pilot to be more on the alert for the possibility of trim misbehavior during the flight. Crisis averted, right?

But no pilot would look at an entry for an AoA sensor being off or worked on, and would think, "Boy, better look out for that safety system I never knew existed that could cause a trim runaway because my previously airworthyness agnostic AoA sensor is unreliable."

Flying and complex system diagnosis in time sensitive conditions requires extensive ahead of time mental model creation ahead of time.

Re:737 Max Explanation by a Software Engineer by lutorm on Sunday March 17, 2019 @03:52PM (#19416128)
It is weird, though, that this was not caught in simulations (not training simulations, but in design simulations.) I would assume that running simulations to see how the design behaves when sensors fail would be an absolutely standard thing to do. I don't work on airplanes, but we routinely simulate sensor failures and their effect on software systems and you'd think this would not be optional for passenger aircraft certification.
Re:737 Max Explanation by a Software Engineer by salawat on Sunday March 17, 2019 @06:00PM (#19416948)
When all of your testing is in house, and you don't have people who are willing to put the project at jeopardy for the sake of doing it right, you'll be amazed what can get overlooked.

If anything, the process of engineering is most difficult in that answering most questions tends to be straightforward (not easy, but straightforward once you know the right methods to apply), while figuring out if you've asked all the right questions is the thing that keeps me awake at night. If you don't ask and dig in until you've answered it fully, it's easy to get blind sided.

I have difficulty believing it could have gone so horribly wrong now, but I can't deny that just from the information available, even if MCAS isn't the root cause of the Ethiopian Air crash, there is some egregious failures in sound practice going on with Boeing to have slipped up this badly.

That's the thing with sound Engineering, when you do it, it just verks. When you don't...

Re:737 Max Explanation by a Software Engineer by CaptainZapp on Sunday March 17, 2019 @02:26PM (#19415462)
Key quote is training on a simulator

That's a tall order when the manufacturer doesn't inform their customers out of commercial expediency.

Re:737 Max Explanation by a Software Engineer by wahern on Monday March 18, 2019 @02:38AM (#19419094)
If you only have two sensors and you're taking input from both sensors, assuming each sensor has the same probability of malfunction then you're doubling the chance of processing bad input.

You could choose to select the input that results in the least-worst bad outcome. But that may not bring the total risk below the increased risk of utilizing both sensors. The sensors are doing something purposeful, after all.

So it may be logical what they're doing. OTOH, the logic may make sense only in a path dependent sort of way, after a series of bad decisions that put them in a corner with poor options. There probably really should be 3 or more sensors.

Re:737 Max Explanation by a Software Engineer by chopin on Monday March 18, 2019 @05:44AM (#19419747)
You take input from both sensors only when they don't disagree (by a margin). There may be failure modes where both have the same bad input but I don't think that this doubles your chances of processing bad input.
Re:737 Max Explanation by a Software Engineer by wahern on Monday March 18, 2019 @01:24PM (#19423048)
You have double the chance that one of the sensors is bad. Which one do you choose? The one that says AoA is normal? Maybe that's the one that is bad. Worse, maybe such a bad input is more likely in a stall scenario.

Without any ability to differentiate bad from good input, you may be better off simply relying on a single sensor at any one time and completely ignoring the other, at least as far as controlling MCAS is concerned. Tossing up a warning that says the sensors disagree is another matter.

Re:737 Max Explanation by a Software Engineer by chopin on Wednesday March 20, 2019 @08:28AM (#19441359)
I meant disabling the system if they disagree.

I agree that one can argue that this doubles the chance of system shutdown compared to properly detecting that one sensor fails. That's obviously only ok if the system is expendable to a certain degree.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Sunday March 17, 2019 @03:37PM (#19416014)
Because there was no available extra sensor. Trevor gets a few other things wrong too.

The elevator feel system (EFS) uses multiple sensors, in fact it has its own pitot probes[1] (two of them) on the elevator. It's not clear if it uses both, but I suspect it also uses both alpha vanes up front like MCAS should be doing. Input from the alpha vane (AoA sensor) is new on the MAX, IIRC. His tweets were a bit ambiguous, but the elevator feel system does not change the trim at all (also EFS was non-op on the Lion Air bird).

It's almost certainly NOT a sensor problem. Look at the graphs from the black box on the Lion Air flight. The angle-of-attack was almost exactly offset by twenty degrees left to right[2]. That sensor was working just fine, but something else on the path was fucked up (bad wiring? bad ADIRU? both have happened before). The fact that the alpha vane was replaced and that didn't fix the angle-of-attack readings supports this. Also, the 737 NG and 737 MAX use identical alpha vanes. While the angle-of-attack data is less important on an NG, we haven't seen people come out of the woodwork talking about how common alpha vane failures are on an NG. I'm inclined to believe that they're not all that common.

The Lion Air flight crew did write up the sensor problems[3].

I saw another one of these "I'm a software engineer so I've got great insight" posts[4] and chuckled. It's a bit of swing and a miss as well. MCAS is hardly silent or subtle. The same trim wheels his Cessna have are present on the 737 and they're noisy[5]. MCAS doesn't quite override the pilot either, and you can see that on the black box graphs from the Lion Air flight[2]. Twenty times the pilot flying entered opposite trim to counteract MCAS and twenty times MCAS paused for five seconds. On the twenty-first time something changed and the plane crashed. It's not quite that simple and that's why there are active investigations.

1: https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAF...

2: https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

3: https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt...

4: https://medium.com/@jpaulreed/the-737max-and-why-software-en...

5: https://www.youtube.com/watch?v=3pPRuFHR1co

Re:737 Max Explanation by a Software Engineer by cameldrv on Sunday March 17, 2019 @11:54PM (#19418585)
I’m the OP Dave BTW. Trevor copied a Facebook post I made a couple of days ago into those tweets. I don’t know all of the details about the EFS system but I’d be pretty sure it doesn’t use data from both sides for anything, because that would be against the brick wall isolation concept in the air data and flight control computers.

You say that the MCAS is neither silent or subtle, but there is no light or master caution or anything, just the trim wheel turning and clicking. I’m not sure you’d notice it if you were wearing a headset and were being distracted by the stick shaker.

The reason they died on activation #21 and not #1 is that the effects are cumulative. Every time it activates it dials in another 2.5 degrees of nose down trim. Eventually you can’t pull on the yoke hard enough to hold altitude.

I agree that the jury is still out as to whether it’s a sensor issue per se or some sort of ADC or data transmission issue.

Re:737 Max Explanation by a Software Engineer by inferiorhuman on Monday March 18, 2019 @12:15AM (#19418657)
The reason they died on activation #21 and not #1 is that the effects are cumulative.

The stabilizer position was graphed on that preliminary report (see: pitch trim position). There was a slight trend towards nose down pitch, but no, at the time they lost control there's a sharp decrease in pitch that doesn't correlate with MCAS. If you look at the "trim manual" line you'll see that the pilot flying was continuing to input nose up pitch on the switches to no effect. Something changed and it wasn't simply that the pilot was overridden by MCAS. After 21 even MCAS continued to counter the pitch up inputs as the stabilizer continued to contribute additional nose down pitch.

I agree that the jury is still out as to whether it’s a sensor issue per se or some sort of ADC or data transmission issue.

To be clear I highly doubt the alpha vane was defective. It's a possibility, but look at that graph. The left "angle of attack indicated" mirrors the right one almost exactly except that there's a near constant twenty degree offset. If that's a sensor problem in two separate sensors I'll pull a Werner Herzog and eat my freaking shoe.

Re:737 Max Explanation by a Software Engineer by engire on Saturday March 23, 2019 @12:42PM (#19471164)
I totally agree here. Im an AME with Max Licence and i would have to say. Im quite sure this wasnt an AOA sensor. The FCC thats active with MCAS as its function (both have this Function) would have been looking at the right vane on one flight and then the left vane on the next flight, however on the last 3 legs the fault was the same. And trust me on this one. May have been different indications bbut all pointed at the same fault. I suspect the FCC or ADIRU was at fault or something it aircraft installation that was inducing the 20 degree difference between the 2.
Re:737 Max Explanation by a Software Engineer by autopilotsw on Monday March 25, 2019 @07:53AM (#19481741)
I don't believe there is a brick wall design here since the FCC outputs are not compared so why would you isolate inputs. I believe it's just a bad design to use the single AOA input.
Re:737 Max Explanation by a Software Engineer by inferiorhuman on Monday March 18, 2019 @10:56PM (#19427686)
You say that the MCAS is neither silent or subtle, but there is no light or master caution or anything, just the trim wheel turning and clicking. I’m not sure you’d notice it if you were wearing a headset and were being distracted by the stick shaker.

BTW the trim wheel has little stripes on it, moves quickly when being operated automatically, and is fairly noisy. It will get your attention. However, my suspicion is that pilots experienced on the NG will mistake the trim wheel doing counterintuitive things for normal STS operation (especially after takeoff). I don't know if the Cessna trim wheel(s) are any quieter or more subtle, but the Cessna certainly doesn't have all the band-aids that a 737 NG or MAX would.

To this point here's a post from a 737 NG pilot[1]:

As a long-time 737 driver I'll just chime-in a few points

[...]

(1) Just after takeoff there is a lot going on with trim, power, configuration changes, and as noted above, the darn speed trim is always moving that trim wheel in seemingly random directions to the point that experienced NG pilots would treat its movement as background noise and normal ops. Movement of the trim wheel in awkward amounts and directions would not immediately trigger a memory item response of disconnecting the servos. No way.

(2) The pilots could very reasonably not have noticed the stab trim movement. Movement of the stab trim on the 737 is indicated by very loud clacking as the wheel rotates. On the -200 it was almost shockingly loud. On the NG, much less so. HOWEVER, the 737 cockpit is NOISY. It's one reason I am happy to not be flying it any more. The ergonomics are ridiculous. Especially at high speeds at low altitudes. With the wind noise, they may not have heard the trim wheel moving. The only other way to know it was moving would be yoke feel and to actually look at the trim setting on the center pedestal, which requires looking down and away from the windows and the instruments in a 'leans'-inducing head move. On the 717, for example, Ms. Douglas chimes in with an audible "Stablizer Motion" warning. There is no such indication on the 737.

[...]

Finally, runaway stab trim is a very, very rare occurence up until now. We trained it about once every other year in the sim because it is so rare. And when we did it was obvious. The nose was getting steadily heavier or steadily lighter with continuous movement of the trim wheel. That is a VERY different scenario than what these pilots faced.

We also trained for jammed stabilizer, the remedy for which is overcoming it with force. The information they were faced with could very reasonably have been interpreted that way, too.

1: https://www.pprune.org/rumours-news/619272-ethiopian-airline...

Re:737 Max Explanation by a Software Engineer by bjowen on Sunday March 17, 2019 @10:28PM (#19418277)
One thing I’ve seen kicking around is that MCAS alternates input between the left FCC and right FCC, which might help understand why the JT43 only had to fight the stick shaker the whole flight (!) but JT610 had the automated trim input as well.
Re:737 Max Explanation by a Software Engineer by m463 on Tuesday March 19, 2019 @07:44PM (#19437379)
serious question: which sensor has failed?

you need at least three sensors. (although the "sensors disagree" option you can purchase is better than nothing)

Re:737 Max Explanation by a Software Engineer by clavoie on Sunday March 17, 2019 @06:00PM (#19416952)
So, a modern aircraft probably have a few thousands sensors (if not way, way more), not strictly redundant but overlapping in some fashion -- I have to believe there's been some modelling trying to check that sensor input is consistent; if only to have some kind of way for the computer to figure out that a sensor has gone crazy (say your compass suddenly detects a significant change in heading, but no sensor has detected even a degree of roll, a change in speed, heading computed by GPS over time is stable... compass might be borked, whaddya say?)

Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input?

Re:737 Max Explanation by a Software Engineer by rocqua on Sunday March 17, 2019 @06:15PM (#19417076)
My guess is that such systems are complex, hard to model for safety assurances, and hard to reason about in the moment.

Yes, it will probably make the system safer, but your guarantees become a lot more fuzzy. This is not the field for fuzzy guarantees.

Re:737 Max Explanation by a Software Engineer by de_watcher on Monday March 18, 2019 @06:45AM (#19420019)
> Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input?

You need to crash a lot of planes to train that NN.

Re:737 Max Explanation by a Software Engineer by m463 on Tuesday March 19, 2019 @07:45PM (#19437389)
there should be at least 3 AOA sensors - so you can not only detect a mismatch, but figure out which sensor is most probably incorrect.
Re:737 Max Explanation by a Software Engineer by cameldrv on Sunday March 17, 2019 @05:52PM (#19416902)
We will see. Part of the avionics design is that there is a “Brick Wall” between the two sets of sensors, air data computers, and flight control computers. If the FCC uses both for MCAS, that safety concept doesn’t exist anymore. That is certainly the reason they didn’t compare the AoA sensors in the MCAS logic. I think they could probably use pitot/static as a cross check on AoA, but it has some tricky edge cases.
Re:737 Max Explanation by a Software Engineer by cmurf on Sunday March 17, 2019 @06:13PM (#19417054)
Reports say it alternates each flight between them; if that's correct, in the case of one good sensor and one bad, that you wouldn't see back to back flights with anomalous behavior related to a bad sensor. Otherwise, perhaps the problem isn't with the sensor itself, but with some part of how sensor information is communicated to the interpretive computer.
Re:737 Max Explanation by a Software Engineer by blattimwind on Sunday March 17, 2019 @02:15PM (#19415393)
AIUI the entire FCC only uses one set of sensors and which set of sensors is used is switched together with the FCC?