News for hackers. Stuff that matters.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by cjbprime on Wednesday March 20, 2019 @03:37AM (#19439815)
This appears to answer the question of why, given that there is a "memory item" for dealing with runaway trim that would have helped, it wasn't followed by the crew:

> “They didn’t seem to know the trim was moving down,” the third source said. “They thought only about airspeed and altitude. That was the only thing they talked about.”

If this is true, not only did the pilots not realize the trim was the cause of the dive and try to fix it, it seems that they didn't even realize the trim wheel was moving.

"Runaway trim" would be something they trained for, but has a presentation so vastly different (of a visibly spinning trim wheel causing quick descent) as to not be recognizably the same emergency to these pilots.

Of course, this crew had no way of knowing that there was a system on the plane that could slowly make erroneous trim changes over a long period with autopilot off, and it makes sense that they might miss it happening while focusing on their stall alerts and speeds and trying to come up with an explanation and also trying to retain control.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by l33tman on Wednesday March 20, 2019 @05:57AM (#19440519)
This is not true, on the released FDR data, it shows clearly that the pilot flying corrected the MCAS system's trim down adjustments at every cycle (each 10 seconds) for most of the level flight. So at least the PF must have been aware of this. It's a bit weird if they didn't talk about it though, maybe it was obvious to both pilots and they were concentrating on if they were keeping altitude and airspeed at all (the most important factors that would keep them flying).
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by mannykannot on Wednesday March 20, 2019 @08:52AM (#19441507)
This got my attention too. For at least 6 minutes, the crew were coping with the problem - the aircraft was holding altitude and speed, if erratically, and they must have been getting the trim back to approximately where it should have been, or else MCAS would have driven the stabilizer to the point where it overrode the elevator authority.

In about the last minute of that recording, the pattern changes: the pilot trim inputs decreased and became much shorter, and the stick forces changed from going up and down to a sustained, high, back-force.

The article says that near the end, the captain handed over control to the 1st officer so he could look for answers. I am speculating wildly here, but could that coincide with the change in the pattern of the pilot's responses? Could it be that the 1st officer did not attempt to re-trim, or did not do it completely? Did he try to trim manually, but perhaps could not because of the load on the stabilizer? [1]. IIRC, according to Dominic Gates, it would only take two cycles of MCAS intervention until the stabilizer had reached a point where full nose-up elevator could not keep the airplane level.

The chart was published in one of Dominic Gates' Seattle Times articles, but I am not in a position to find it right now...

[1] https://news.ycombinator.com/item?id=19400157

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by VBprogrammer on Wednesday March 20, 2019 @09:07AM (#19441633)
Anyone who's ever piloted an aircraft would try to re-trim against pitch forces. It's inconceivable that the pilot for some reason just failed to do that. I'm not sure exactly what force it takes to get full nose up elevator at flying speed but it will be in the range of dozens of kilograms.

I personally wonder if there is potentially a thermal cut-off on the trim motor which could have been activated by this trimming and re-triming. Or even an outright failure of some part of the system.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by VBprogrammer on Wednesday March 20, 2019 @11:19AM (#19443037)
I just had a better look through the graphs in the report and I can see exactly what you are talking about. There is a clear pattern of trimming against the MCAS, then just before the speed increases and the altitude decreases the pattern changes. There are still nose up trim commands but much shorter. At that point the pitch trim position changes and never recovers to the same point where the aircrafts height and speed had previously been relatively stable.

Also notable is that engines spool-up about the same point to what looks like full takeoff thrust.

They must have identified an issue just after going flaps 0 because there is no other reason to go to back to flaps 5 (which incidentally disables MCAS). Why on earth they didn't drop flaps again and head back to the airport is a bit of a mystery to me.

I can't imagine the Captain handing control of a aircraft which was obviously a handful to a co-pilot and then looking straight down at the checklist, surely he'd have given guidance about the trim and waited to ensure the co-pilot was handling the aircraft?

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by dvdkhlng on Wednesday March 20, 2019 @06:31AM (#19440688)
I'm just wondering: it looks like the absolute trim angle is not visible anywhere in the cockpit? The trim wheel takes multiple revolutions to adjust trim between min/max limits, so looking at the wheel itself, you can only every see changes, not values. Unless you notice the change while it happens, you won't have any indication as to the unusual trim angle?

Thinking about that I wonder whether there was even a measurement signal that the computers could rely on to determine the current state of the trim angle in absolute terms.

Given how the MCAS did incremental changes without stopping before reaching the mechanical limits, I do imagine that there might have been no hardware feedback to assess the current trim state, thereby making it impossible hardware-wise to limit MCAS to a sensible (and safe) range of angles.

[edit] Looking at the lion air crash report [1] p.14 ff it seems like also the black box only recorded trim changes, not absolute trim state.

[1] https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by VBprogrammer on Wednesday March 20, 2019 @07:36AM (#19441038)
There is a trim position indication. It's to the left and right of the throttles. It's a critical thing to check before starting the takeoff roll.

You can see it on the copilot's side here by the trim wheel.

http://www.b737.org.uk/images/stabtrim_200.jpg

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by dvdkhlng on Wednesday March 20, 2019 @08:02AM (#19441205)
Yes, thinking about that, not having any pilot-visible indication would be quite some point of failure at lift-off.

Still wondering whether MCAS and other automatation systems have a representation of that position indication. I can't currently come up with an idea of how to implement that in an easy, yet error-resilient way. Usually you would initially drive into a reference position, then integrate the motor's actions to arive at an absolute value, but integration would potentially add up subsequent errors, and obviously you can't re-do the referencing step while in the air.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by VBprogrammer on Wednesday March 20, 2019 @10:54AM (#19442802)
FYI It does look like the graphs here indicate "pitch trim position" which I believe is the absolute value.

https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by VBprogrammer on Wednesday March 20, 2019 @08:08AM (#19441238)
A potentiometer (or something fancier) directly attached to the stabilizer would do it. I'm not sure if they do have one or not. They rate of movement on the stab trim is different depending on the source of the trim change so it's possible they have a PWM controller with some feedback.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by fgonzag on Wednesday March 20, 2019 @03:35PM (#19445976)
Or just use an angle sensor on the control surfaces? You now have two measurements: What the computer is telling the hardware to set the angle to, what the angle actually is.

These two are very different but very important pieces of information.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by coredog64 on Wednesday March 20, 2019 @09:10AM (#19441672)
Earlier versions of the 737 (up to next gen) have noisy trim wheels that are difficult to ignore.

AIUI, runaway trim has always been a possibility and the trim wheel design helps to surface that it’s happening.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by RidingPegasus on Wednesday March 20, 2019 @03:53AM (#19439892)
Was the trim wheel not visibly spinning out of their control? Simply because the symptoms are different should mean that stopping it via taking manual control is a recurring option.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by crooked-v on Wednesday March 20, 2019 @04:06AM (#19439945)
Part of the problem is that the MCAS system has a 20-second delay after the last time the pilot touched controls for the flaps, and it adjusts itself to a new position and then stops. There are no other indicators that it's happening, and since it's noncontinuous and has a delay, it would be easy to miss the only visual sign of the trim wheel readjusting itself once, especially since the wheel itself is by the pilot's thigh outside of forward line of sight.

In a normal runaway trim situation, by comparison, the trim wheel continues obviously spinning as long as the problem is going on, so it only takes a single glance at any time to notice it.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by oasisbob on Wednesday March 20, 2019 @06:07AM (#19440557)
I'm not sure that's accurate. Other descriptions of the MCAS system state that it will continually modify the trim on the horizontal stabilizer for up to 10 seconds at a time.

http://www.b737.org.uk/mcas.htm

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by usrusr on Wednesday March 20, 2019 @09:21AM (#19441813)
Well, the point is that the experience from MCAS trying to be clever and subtle is very different from what one would expect in an old-school runaway trim situation caused by, say, a stuck relay.

Easy to chase down a wrong line of inquiry when the symptoms only appear sporadically and can be temporarily stopped by a whole range of actions that won't permanently fix it.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by inferiorhuman on Wednesday March 20, 2019 @06:20AM (#19440630)
To summarise; MCAS will trim the Stabilizer down for 10 seconds (2.5 deg nose down) and pause for 5 seconds and repeat if the conditions (high angle of attack, flaps up and autopilot disengaged) continue to be met. Using electric pitch trim will only pause MCAS, to deactivate it you need to switch off the STAB TRIM SUTOUT switches.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by verytrivial on Wednesday March 20, 2019 @06:04AM (#19440544)
The Lion Air directive has instructions on page 41 of their preliminary report that include:

5. If the runaway continues after the autopilot is disengaged:

   STAB TRIM CUTOUT switches (both)  ... CUTOUT
   If the runaway continues:
      Stabilizer trim wheel  .... *Grasp and hold*
This may be an overabundance of caution, or to cover other runaway events, but it reads like "you may need to physically fight the computer even when it has been disabled". Not exactly heartening.

EDIT: A comment below points out haptic feedback to disengage as a reason to grab-and-hold.

https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by ProAm on Wednesday March 20, 2019 @11:47AM (#19443356)
> you may need to physically fight the computer

This is the future I look forward to. AI/ML/Autopilot/Companies moving fast and breaking things.... Boeing took that one literally.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by mlindner on Wednesday March 20, 2019 @06:34AM (#19440711)
That's incorrect, these have haptic feedback, once you grasp it, it's not going to fight you. It looks for resistance and if it sees the resistance it doesn't continue.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by userbinator on Wednesday March 20, 2019 @08:21AM (#19441324)
Really? Here's a video of someone doing it, and it certainly looks like it tries to fight you (and the trim motor continues running) but there's a clutch that starts slipping:

https://www.youtube.com/watch?v=cQirIH_DuAs

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by heavenlyblue on Wednesday March 20, 2019 @06:53AM (#19440822)
You are still fighting it, aren't you?
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by verytrivial on Wednesday March 20, 2019 @06:50AM (#19440800)
Ah, Okay, thanks. What remaining automation would be driving the trim wheel in this case that grab-and-hold would countermand?
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by cjbprime on Wednesday March 20, 2019 @12:22PM (#19443742)
I think it's definitely to cover other runaway events, e.g. mechanical failure. (And I don't think there's haptic feedback.)
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by frenchy on Wednesday March 20, 2019 @04:03AM (#19439933)
I doubt it would have been "out of control", it would have just been slow changes over time as trim increased. The pilots might not have been fighting it much either, if they thought that airspeed was the problem, and not pitch.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by Someone1234 on Wednesday March 20, 2019 @04:10AM (#19439968)
The graph on page 14 of this report[0] is pretty telling about how slow the changes were... Note Trim Manual, Trim Automatic, and Pitch Trim Position.

[0] https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by acqq on Wednesday March 20, 2019 @04:42AM (#19440133)
Thanks! Page 23 of the PDF or 14 as printed at the bottom. Sadly it's hard to compare with the previous flight (pg. 25 or 16) as for it they published the graph of the whole flight instead of only the part until the (at the time still secret) MCAS was turned off (or probably more correctly, the motor controlled by the MCAS, which actually steered the "trim").
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by cjbprime on Wednesday March 20, 2019 @04:01AM (#19439919)
The spin was not visible to these pilots presumably because it was happening in small increments with a delay between them.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by ahoka on Wednesday March 20, 2019 @04:24AM (#19440045)
If anyone wonders this is how it looks like: https://youtu.be/-X01uY0bJs0?t=48
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by TheSpiceIsLife on Wednesday March 20, 2019 @05:01AM (#19440224)
And sounds?

The trim wheels moving also makes a sound?

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by WalterBright on Wednesday March 20, 2019 @05:42AM (#19440427)
Yes. The ratcheting sound is deliberate, so the pilots don't need to see the wheels spin to know the trim is running. Also note that there's a white stripe on the wheel so it's easy to see it is running.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by jacobush on Wednesday March 20, 2019 @09:18AM (#19441777)
Someone else in the thread said newer models don't make a sound.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by michaelt on Wednesday March 20, 2019 @05:48AM (#19440464)
You'll note in that video, the left-hand pilot has a headset on, presumably the right-hand pilot does too. Headsets are seen in other videos [1] of the cockpit.

So even if the spinning trim wheels make noise, they may not have heard it.

(My google image search shows some pilots wearing two-ear headsets, some wearing one-ear headsets, some wearing two-ear headsets with side not on their ear, and some not wearing headsets. But some of those are pilots posing for photos, and presumably pilots don't let people visit the cockpit during critical stages of the flight)

[1] https://youtu.be/wz2Q3COfveg?t=409

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by chrisseaton on Wednesday March 20, 2019 @06:24AM (#19440653)
I don't understand do you think they're listening to music or something?

Having a headset on doesn't stop you hearing noises in a cockpit. How did you think all the other verbal alarms and callouts worked if they can't hear anything?

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by TheSpiceIsLife on Wednesday March 20, 2019 @07:05AM (#19440875)
I was thinking the same thing.

I wear earplugs all day in a metal fabrication workshop.

The earplugs certainly don’t prevent you from hearing anything.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by vultour on Wednesday March 20, 2019 @07:08AM (#19440888)
Those all go into the headset as well AFAIK. Many pilots use noise cancelling headphones.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by chrisseaton on Wednesday March 20, 2019 @07:26AM (#19440975)
If the deliberate clack of the trim wheel also goes into the headphones then why are we having a discussion about whether headphones stopped them hearing the deliberate clack of the trimwheel?
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by alistairSH on Wednesday March 20, 2019 @09:14AM (#19441732)
My guess is the electronic beeps and boops could be piped into the headsets, but the mechanical clack-clack of the trim wheel is not.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by moftz on Wednesday March 20, 2019 @09:02AM (#19441591)
Noise cancelling headphones would still let you something clicking right in front of you. In fact, open-back noise cancelling headphones would be better to wear than something that just muffles the noise if you need to hear your environment but just want to cut out the hum of the engines.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by cblades on Wednesday March 20, 2019 @09:28AM (#19441870)
I'd imagine that that would be less common (or at least, they would use headphones with a lesser effect) with commercial jets. They're far enough from the engines that the noise should be pretty tollerable.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by ekianjo on Wednesday March 20, 2019 @05:38AM (#19440410)
From the videos I have seen in cockpit it does seem to make a sound as it rolls.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by acqq on Wednesday March 20, 2019 @04:04AM (#19439940)
Exactly. It was not "runaway" in the sense of "continuously doing the same." Something was happening, from the point of view of the pilot and co-pilot "at random." (Not to mention that on the previous models exactly such pattern was effectively impossible to happen). The faulty non-redundant sensor was producing false reading and that was the input to the to the pilots completely unknown MCAS which then moved the nose down based on the difference between the input and MCAS' expected target, turning "the correction" on only from time to time.

The presence of the third, off-duty pilot on the flight before additionally explains even better how in that case the crash was avoided. While the pilot and co-pilot were at the controls, the third one, looking from behind, didn't have the direct duties to do, so he had enough time to both just observe everything that was going on and the less immediate stress to allow him to come to the solution that worked.

Like when one programmer spends half an hour on something "not working" and complains to the colleague who just takes a look and immediately sees what the first one hasn't for half an hour. Additionally the movement of the trim wheel is actually easier to follow from the place in the back (see https://news.ycombinator.com/item?id=19440045 ).

And there's also a possibility that the false sensor reading and MCAS together produced a slightly different differential signal which resulted in more obvious manifestation of the problem. Possibly both elements contributed to the lucky outcome in that previous flight.

In the old times of civil aviation there was the third person on duty in the cockpit, the "flight-engineer." As the planes got more computerized controls, they started to be certified to fly with two-pilot crews. The computerized systems, when working, do reduce the chance of the crash and overall number of accidents did decrease simply because the humans were on the average less often in control, and therefore there was less chance to do anything wrong.

The problem is when the unreported computerized controls secretly depend on the single sensor that can be faulty. And when they actively confuse the pilot and ruin the flight instead of helping him.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by kayfox on Wednesday March 20, 2019 @12:59PM (#19444152)
"Runaway" in the sense of "its moving in a undesired and unexpected direction and continues to do so despite attempts to stop it".
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by jeremysmyth on Thursday March 21, 2019 @04:07PM (#19456691)
"Runaway" is a term of art here. Redefining it for this conversation is unhelpful.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by dukoid on Wednesday March 20, 2019 @04:41AM (#19440130)
To my understanding, the trim wheels are moving frequently for small adjustments -- and it's possible to switch the wheel movement off if this becomes too annoying.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by VBprogrammer on Wednesday March 20, 2019 @06:32AM (#19440694)
It's running for 10 seconds at a time for 2.5 units of trim out of a possible ~15. That's quite a big trim change. I find it hard to believe that it's not noticed because of the 'boiling a frog' situation.

It not possible to disable wheel movement. It's mechanically linked to the jackscrew.

Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by losvedir on Wednesday March 20, 2019 @04:46AM (#19440145)
Also not a pilot but I don't think you can switch the wheel off. You can, however, disable the electric motors that spin the wheel, which is what the checklist for runaway trim tells you to do, and what was supposed to have been done here.
Re:Cockpit voice recorder of Lion Air jet depicts pilots' fr... by WalterBright on Wednesday March 20, 2019 @05:44AM (#19440441)
The spinning wheels are directly connected to the stab trim via cables. If the trim motors fail, the pilots can turn those wheels by hand (there's a folding crank on them) to manually adjust the trim.