News for hackers. Stuff that matters.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by rocqua on Monday February 04, 2019 @09:31AM (#19076582)
One interface change was the effect of 'pulling hard back on the stick' in case of runaway stabilizers. That worked with the old system, but not with MCAS.

This seems to be exactly the interface change that lead to the crash.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by CydeWeys on Monday February 04, 2019 @09:39AM (#19076645)
To use a car analogy ...

You can always override cruise control by stomping hard on the brake (like to avoid an imminent crash). That's how it's always worked and you've gotten used to this, and done it on occasion when warranted.

Now imagine that the next generation of adaptive cruise-control/"auto-pilot"/whatever comes out, and stomping on the brakes no longer does anything. You have to first disable the cruise control by pressing a button on the steering wheel, and only then will inputs to the brake pedal do anything.

And then you don't tell drivers about this change.

You can totally see how, right in the lead-up to a fatal accident, a driver is going to be focused solely on stomping on that brake pedal in increasing panic, wondering right up to the moment of death why that's not doing anything. They won't consider the cruise control off button because it's not their most immediate need (braking is), and they've never needed to use it before.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by del82 on Monday February 04, 2019 @10:08AM (#19076877)
But the difference is not only do the pilots of a commercial airliner have vastly more training and experience than your average driver, but they literally have a list of things to try (and extensive training on following the list) in the case of a malfunction like an uncommanded nose-down.

Imagine that the failure that caused the nose-down wasn't a failed AOA sensor giving bad readings to MCAS, but some other reason that _also_ wouldn't have been solved by pulling back on the yoke, but would have been by completing some later step on the checklist. Suppose the pilots didn't follow the checklist. Would that be enough information to say that Boeing is responsible?

To be sure, in this case the pilots may have followed the checklist! It may well be the case that Boeing is completely responsible! The checklist items might not have worked, or there may have been a good reason that the pilots didn't follow it, or the checklist might have been crazy, or they might not have had time to do what needed to be done, etc. There's still a whole lot that's not known (or hasn't been released) about what happened.

I'm just not sure that the current evidence, _viz_ that Boeing made an internal software change, that they didn't explicitly call it out to pilots, and that there's no difference in the actions prescribed in the event of an uncommanded nose-down pre- and post-change, is enough to say that the fault is entirely Boeing's for this accident.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by llukas on Monday February 04, 2019 @10:26AM (#19077021)
More training and experience is a problem as previous plane model behaved differently. Pilot trained this situation and plane did X and now does Y - you don't really see the problem? - now pilot has extra burden to quickly determine if something else is wrong as plane behaves differently than pilot was trained to.

And you are writing this after many, many accidents that root cause was pilot not exactly knowing what or why something happens with a plane or plane autopilot (ie. AF 447)

Being apologetic of cost cutting on safety issues is dumb as it erodes culture of safety and encourages others to skimp on safety.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @11:00AM (#19077356)
> Pilot trained this situation and plane did X and now does Y - you don't really see the problem? - now pilot has extra burden to quickly determine if something else is wrong as plane behaves differently than pilot was trained to.

The pilots did not train for a specific root cause of a fault. They trained for a symptom (uncommanded nose down), and the procedure for that situation was unchanged.

> extra burden to quickly determine if something else is wrong

This isn't an extra burden. Pilots aren't doing root cause analysis for failures while they are responding to them. They are trained to try actions in a specific order until something works. It's not like the checklist used to have one action on it and now it has two - the solution in this case was a standard action on the standard checklist.

Pilots do not look around, say "ah, electrical short in elevator actuator" or "ah, bad angle of attack sensor" and then take a single action.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by llukas on Monday February 04, 2019 @11:08AM (#19077435)
They did train recovery scenario on simulator and know how plane should behave (based on old model knowledge) - now it behaves differently. All you wrote is they should ignore it as not important detail and stick to the checklist.

Unexpected situations are always extra cognitive burden.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by ansible on Monday February 04, 2019 @11:40AM (#19077725)
> Unexpected situations are always extra cognitive burden.

From my perspective, it is a human factors issue that Boeing failed to consider.

Yes, the emergency procedure remained the same, but even well-trained pilots are still people. And changing the behavior of the system in such situations was not well-advised.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @10:53AM (#19077279)
I do not think this is a good analogy.

Firstly, there is no single equivalent to "slamming on the brakes" for uncommanded nose-down. This could be caused by a variety of faults, and pilots are trained to respond in a fashion that will be effective for even those in which the first thing to try doesn't work. There is a standard procedure in place to use in this type of situation - arguing that the pilots should only be expected to do one thing with increasing desperation is essentially arguing that they will not be able to respond to a whole host of emergencies causing uncommanded nose-down.

Second,

> Information from the flight data recorder shows that the plane’s nose was pitched down more than two dozen times during the brief flight, resisting efforts by the pilots to keep it flying level ... The standard checklist for dealing with that sort of emergency on the previous version of the 737 focuses on flipping the stabilizer trim cutout switches and using the manual wheels to adjust the stabilizers. [emph mine]

your argument essentially hinges on the assumption that pulling hard back on the stick is a sufficient solution for all the problems that may happen with a plane with the exception of a fault with MCAS (the new system). I don't think that is accurate. Even prior to the the 737 max! It sounds there were a lot of things that would require further action that pulling back on the stick.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by CydeWeys on Monday February 04, 2019 @11:01AM (#19077371)
Their response would have worked to return to level flight in non-MAX variants of the plane, though. Regardless of what the manual or checklist says, they had years of experience flying 737s that behaved in a specific way, they developed an unconscious/intuitive mental model of the plane based on those behaviors (that is faster and quicker to react than consulting checklists), and then a significant and deadly change was made to those behaviors and not communicated to them for cost-cutting reasons.

There's clearly a problem on Boeing's end here.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by poof131 on Monday February 04, 2019 @11:30AM (#19077634)
From the article:

"Older 737s had another way of addressing certain problems with the stabilizers: Pulling back on the yoke, or control column, one of which sits immediately in front of both the captain and the first officer, would cut off electronic control of the stabilizers, allowing the pilots to control them manually.

That feature was disabled on the Max when M.C.A.S. was activated — another change that pilots were unlikely to have been aware of. After the crash, Boeing told airlines that when M.C.A.S. is activated, as it appeared to have been on the Lion Air flight, pulling back on the control column will not stop so-called stabilizer runaway."

If the above is true, it's near criminal that Boeing didn't notify pilots about the change. It's also not just about following checklists but understanding the systems of the aircraft.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by cmurf on Monday February 04, 2019 @02:52PM (#19079979)
There's a Faustian bargain made by accepting so much abstraction from the true nature of an airplane by using software, and yet also that abstraction is not absolute. Pilots are made responsible for knowing all of the aircraft's behaviors, with full abstraction (normal mode), as well as partial abstractions (error and failure modes, of which there can be quite a few for a specific made/model, many of which must be inferred rather than being clearly displayed). If pilots get confused, however complicated and unintuitive the system behavior, they are blamed.

The more "piloting" computers are doing, it seems appropriate that they will be increasingly, properly, accused of the equivalent of "pilot error". And yet here is Boeing, taking more and more piloting responsibility, while still blaming the pilots when that doesn't work out. It's a deification of engineering: when things go properly and safely, praise engineering; when things fail, blame the pilots.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @11:20AM (#19077538)
The point is that their response may have worked for this particular fault, but would not have worked in general. There is a reason that there are more procedures than "pull back on the stick" - pilots do not know what the fault is.

> they developed an unconscious/intuitive mental model of the plane

If the plane has a fault, it's frequently not going to behave like their unconscious model says it should. In the happy case, rely on your intuition. In the unhappy case, when things are working as they should, follow the emergency procedure.

Sorr, but this is a little like saying "follow standard procedure to put the landing gear down. If that procedure doesn't work, repeat your intuitive action over and over until hopefully the landing gear goes down." No - follow the emergency procedure in place for landing gear fails to descend.

> not communicated to them for cost-cutting reasons

Take it up with the FAA and the airlines?

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by poof131 on Monday February 04, 2019 @11:33AM (#19077666)
> Take it up with the FAA and the airlines?

Sure. It's not just Boeings fault. I believe the FAA will course correct from this and probably all changes to flight controls will need to be reported to pilots. More lessons written in blood.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by CydeWeys on Monday February 04, 2019 @01:56PM (#19079422)
It's still more Boeing's fault than anyone else's though. They were the ones who made this change and then tried to hide it to the fullest extent possible, so that it wouldn't trigger mandatory retraining.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @03:26PM (#19080289)
> and then tried to hide it to the fullest extent possible

I haven't seen any evidence of this. It's definitely not supported by this NYTimes article.

Do you have a source?

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by callmeal on Monday February 04, 2019 @11:12AM (#19077472)
>Firstly, there is no single equivalent to "slamming on the brakes" for uncommanded nose-down.

Actually in the "old" version, there is.

From TFA:

Older 737s had another way of addressing certain problems with the stabilizers: Pulling back on the yoke, or control column, one of which sits immediately in front of both the captain and the first officer, would cut off electronic control of the stabilizers, allowing the pilots to control them manually.

Which makes the car analogy apt.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by richardhod on Monday February 04, 2019 @12:20PM (#19078267)
Indeed, any pilot will respond to nose down with ,pulling back on the yoke, or on the joystick, in other and older aircraft. This is a basic flying control, as basic as steering wheel and brakes, making this an excellent analogy. Any system that complicates this in the way described is a terrible system, for the reasons explained above.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by loeg on Monday February 04, 2019 @12:09PM (#19078107)
Drivers aren't trained to follow checklists and usually don't have dozens of seconds to respond to mechanical emergencies. Cars also don't fall out of the sky if they break down. It's not a great metaphor.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @12:39PM (#19078502)
> have dozens of seconds to respond

12 minutes, in this case.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by loeg on Monday February 04, 2019 @02:40PM (#19079877)
Yeah, I figured it's usually many minutes but didn't want to underplay the stress and difficulty of following a checklist in an extreme emergency. In contrast, car major mechanical failures generally resolve themselves extremely quickly.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @11:35AM (#19077675)
And yet the pilots of the previous flight flipped the cutout switches.

EDIT: ... Presumably because it's literally the second thing on the list of things to do in the case of a runaway stabilizer.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by AcerbicZero on Monday February 04, 2019 @02:19PM (#19079682)
I strongly agree. I've read most of the other comments here, and as an armature pilot myself, I think most of the responses to you are missing the point.

Its extremely unfortunate, but these pilots had ~10 minutes of flight time between their request to return to the airport noting aircraft control issues, and their crash, during which they completely failed to follow their checklists. Lion Air put those under-trained (if we're being generous) pilots into that cockpit and they are the only ones who carry responsibility for this crash.

Its anecdotal, but most of my older flight instructors have been lamenting the loss of critical pilot skills in the industry, and this appears to be just another example of that.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by ridgeguy on Monday February 04, 2019 @01:30PM (#19079118)
>your argument essentially hinges on the assumption that pulling hard back on the stick is a sufficient solution for all the problems that may happen with a plane with the exception of a fault with MCAS (the new system).

I don't read it that way. To me, it's specifically based on removal (without notice to pilots) of a function that would absolutely have terminated a trim runaway condition.

In other words, Boeing eliminated one well-known and trained-on trim runaway fix, and didn't tell the pilots. This is different from saying that stick pullback is a universal solution.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by FabHK on Monday February 04, 2019 @10:04AM (#19076840)
I think that's a pretty good analogy. If you do everything right, you'll get this malfunction under control. But it's easy to see how pilots whose plane is acting up unexpectedly can miss it, and it's reasonable to ask whether Boeing could've done better - and the regulators (note that according to the article, FAA and EASA were convinced by Boeing that this did not require re-training, while the Brazilian regulator wasn't).
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by del82 on Monday February 04, 2019 @10:16AM (#19076939)
> If you do everything right, you'll get this malfunction under control. But it's easy to see how pilots whose plane is acting up unexpectedly can miss it, and it's reasonable to ask whether Boeing could've done better

Absolutely reasonable to ask what Boeing could have done better. I'm just not sure the information about Boeing's actions contained in the article gets me all the way to "indefensible", paraphrasing another comment.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by LoSboccacc on Monday February 04, 2019 @11:39AM (#19077713)
> That's how it's always worked and you've gotten used to this

but still, instead of going trough the full checklist they stopped and tried repeatedly whatever fixed the issue in the past/on the simulator.

a more complete analogy:

"follow these ten step do diagnose a bug on the software"

"but last time it was just a compilation flag, I'll check the compilation flags"

"bug persists"

"last time it was just a compilation flag, I'll check compilation flags"

"bug persists"

"last time it was just a compilation flag, I'll check the compilation flags"

"system halts"

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by crooked-v on Monday February 04, 2019 @12:48PM (#19078624)
From what I understand, part of the problem was that it helped temporarily... and then the system pointed the nose down again.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by LoSboccacc on Monday February 04, 2019 @01:58PM (#19079445)
yeah I'm not siding with Boing here, what I'm saying is that pilots are given checklists for a reason, and going by the usual hunches instead of following the checklist as they were supposed to is as much a failure in training as is a failure in the plane user interface.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by FabHK on Tuesday February 05, 2019 @09:30AM (#19085536)
While the trim is not too far down, you can still counteract it with elevator (pulling the yoke). But then, MACS kicks in again and pushes the trim further and further down, and if you don't trim up and/or switch the trim cut-out, then you get into the situation where you can't recover anymore using the yoke.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by acklenx on Monday February 04, 2019 @12:53PM (#19078675)
"system halts"

That's one heck of an analogy! Brutal

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by cmurf on Monday February 04, 2019 @01:51PM (#19079365)
Or that the system fights your stomping hard on the brake by accelerating. In the Lion Air crash, the system fought the pilots by aggressively nosing down. That it was doing this because of faulty sensor data, is different than how other 737's behave in the same situation.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @11:46AM (#19077787)
For reference, here is the runaway stabilizer memory item (the "checklist") for 737:

1. Control column ............................. Hold firmly

2. Autopilot (if engaged) ..................... Disengage

3. IF the runaway stops:

------------------------ [done]

4. IF the runaway continues:

STAB TRIM CUTOUT switches (both) ...... CUTOUT

IF the runaway continues:

Stabilizer trim wheel ............ Grasp and hold

EDIT:

It's #4 that's of interest here. People saying that the interface changed are saying that it's fine if pilots stop after #1, even when dealing with runaway stabilizer for 12 minutes.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by Someone1234 on Monday February 04, 2019 @12:03PM (#19078037)
Accurate but irrelevant. This assumes the pilots knew they were dealing with a runaway stabilizer, and considering M.C.A.S. wasn't behaving similar to their previous training/experience/simulations with Runaway Stabilizers, it isn't clear they'd know they should follow this checklist. Now had they been actually trained on M.C.A.S. inc. faults, they may have known to do exactly this and we wouldn't even be discussing it.

This has been discussed in great detail on other flight forums.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by twtw on Monday February 04, 2019 @12:07PM (#19078095)
It's not really possible to not know that you are dealing with a runaway stabilizer. MCAS (and every other automatic system to adjust trim) causes large physical wheels at the side of the pilot's knee to spin.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by sokoloff on Monday February 04, 2019 @12:40PM (#19078524)
And these large wheels have small bell/clackers on them so you get a distinctly audible signal in addition to the large black wheels with white stripes on them.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by sokoloff on Monday February 04, 2019 @02:55PM (#19080013)
Here's a video of the wheels in motion: https://vimeo.com/34501723

Disregard the text on the video description as it's blatantly wrong (describing that the trim tabs still move without the wheels moving; wrong on two accounts: the trim doesn't move without the wheels moving and the 737 uses a jackscrew for horizontal stabilizer trim rather than trim tabs [which is why the pilot can't simply override the aerodynamic force as they could with a trim tab])

Jackscrew operation: https://www.youtube.com/watch?v=rxPa9A-k2xY

The 7-3 has balance tabs to make the control forces lighter in the event of a hydraulics failure, but these are not trim tabs in any sense of the word.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by rocqua on Monday February 04, 2019 @03:55PM (#19080538)
As I read the article, it seemed like before, step 1 would suffice before. If this is not the case with MCAS, it could at the very least throw the pilot's diagnostics. "Is it runaway stabilizers? Well, no response from holding the control column firmly so I guess not. Lets check other things".

Sure, the correct response is to follow the checklist, but should we really rely on pilots always knowing the correct response? Especially when that goes against their previous experience? Not that I am defending these pilots not knowing the checklists, instead I am arguing we should take into account learning from experience rather than theory.

Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by del82 on Monday February 04, 2019 @09:38AM (#19076639)
That's fair, but what does the checklist say? I mean, say there are a thousand different malfunctions that could cause an uncommanded nose-down. This one happens to be #359, but the pilots don't know that, they just know that they're pointed at the ground. Maybe it used to be the case that the first item on the checklist (pull back on the stick) fixed problem #359, and now it's the second. But there are several hundred other malfunctions that might have caused the problem that also aren't solved by pulling back on the stick, so the next move is to go to the next item on the checklist, right?
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by peteradio on Monday February 04, 2019 @09:57AM (#19076791)
Pull back wasn't in the original manual but was a relied upon method. Boeing fucked up years ago by not documenting the function so it was allowed to be dropped transparently. How is that defensible?
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by lmm on Monday February 04, 2019 @11:30AM (#19077639)
Why was a method that wasn't in the official checklist "relied upon"? Pilots following some undocumented, non-standard procedure sounds like their fault rather than Boeing's.
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by peteradio on Monday February 04, 2019 @12:17PM (#19078210)
Who knows? Have you ever been taught an undocumented procedure by the expert and been told to use it regardless of what the manual states? Happens all the time? Is a pilot in a position to affect Boeing beauracracy?
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by lozenge on Monday February 04, 2019 @02:06PM (#19079535)
Sorry but how could Boeing design any plane if they can't assume pilots will follow the manual or any checklists?
Re:Behind the Lion Air Crash, a Trail of Decisions Kept Pilo... by rocqua on Monday February 04, 2019 @03:58PM (#19080561)
By including features in the manual. The alternative is that they don't, people rely on the features, they change the features without notifying anyone, and when shit goes wrong they can say "It wasn't documented: shouldn't do that".

The real question here is how commonly this feature was used. If it was common, then not putting it in the manual is a big problem!