News for hackers. Stuff that matters.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by zaroth on Friday November 08, 2019 @09:05AM (#21482192)
The can of worms has been officially opened, and this may turn out to be a case study in how bad software and outsourcing your centers of excellence can bankrupt a company.

If TFA is to believed, Boeing has embarked on a project to take what is currently a master-slave failover design and jury-rig it into a master-master real-time system. This is based on bad performance in the simulator—not because of MCAS—but while testing errant memory errors from a simulated gamma ray strike!

The obvious question of how can dual-master ever work when they disagree if there isn’t a 3rd source of truth to vote out the failure? This means the software running in both computers has to somehow agree which one is wrong, while one of them is in a potential failure state. This... isn’t how flight control systems are designed from first principals.

I’m not a flight system designer, but there must be hundreds of physical, electrical, and architectural considerations taken at every point in the design process which enable multi-master controllers with the ability to vote out a failure. Everything from the particular sensor suite, the number of sensors, the way they are wired, the way data is acquired and bused through the system, the timing and synchronization of the system clocks, the way that control outputs are calculated, queued, and ultimately issued to downstream controllers... none of the necessary pieces will be in place in a system which up until now makes you select a single master controller before embarking on your flight.

The only conclusion I can reach is that Boeing has lost its mind, and this project is absolutely doomed to fail. The architecture that Boeing has apparently committed itself to now is extremely difficult to design from first principals and a blank slate. I just don’t see how it’s something that can realistically be papered onto a legacy dual-computer system after the fact.

The part in TFA about “adding a wire” practically made me spit out my coffee. The sheer level of arrogance that Boeing management must have to think this would be possible - is just a classic example of an elitist MBA management group totally disconnected from any technical domain expertise. Boeing said they would have this ready for certification by the end of this year?!

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by drawkbox on Friday November 08, 2019 @12:01PM (#21484163)
Classic case of hitting the date but crunched out with technical debt and buggy software, a tell that management/finance/marketing is driving the decision over the engineer/designer/product people.

People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.

MBA/business and marketing need to go back to letting the product people and engineers create value, then they can extract the value and sell it. It is much easier to sell value of a good product, it basically sells itself. For some reason American business has completely forgotten the rule this millennia, product then marketing and selling.

You see this same thing happen at game companies when they put a date out there and it is driven by management and marketing over reality of engineering/product/design that makes a fun and solid game.

Boeing was greatly diminished in engineering/product excellence by the McDonnell Douglas management control takeover, everything newly produced since has the tell of this.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by joe_the_user on Friday November 08, 2019 @03:46PM (#21486534)
People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.

The problem is the 737 Max was hustled quickly to market to beat airbus and especially to offer a "drop in replacement" to existing earlier generation 737. Boeing would have definitely lost market share if they hadn't gotten their plane out then.

Choosing technical debt involves economic logic for today's marketplace. I suspect the real problem is companies wind-up deluded by their choice - to hustle the 737 Max to market, Boeing had to pretend is wasn't a, uh, piece of crap. Or not a piece of crap but a terrible mix of high tech and low compromises. And this pretending had to go through the whole enterprise. The marching order couldn't be "we'll add few more stiches on this Frankstein and hope the guts don't spill out", they had to be "this will be one more product of Boeing excellence" and everyone either believes or acts like they do. and this situation, combined with the huge investment involved, makes cutting losses very hard.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by nikanj on Friday November 08, 2019 @02:17PM (#21485706)
> People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.

#notallMBAs, but the surest way for your product to fail on the market is for your company to go bankrupt before you’ve polished it enough.

In this specific case it was about greed, but sometimes MBAs push for release, because according to the engineering teams the product is never truly perfect

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by drawkbox on Saturday November 09, 2019 @04:24PM (#21494461)
A crunch type culture and pushing can also create the issues seen in these problem projects. In fact I personally believe this is why more software and products are bad today, the product professionals are being overridden by marketing/finance and exact dates to meet some marketing goal not a good product goal.

Product people and engineers know how to manage time probably better than business because they are doing the work.

If something fails because it doesn't meet the market there are numerous reasons for that besides not hitting the date.

A good product, that hits the date, can still fail due to market timing or market movement and many other things. Whether a project is launched at a certain date has little bearing on the success of that project long term, short term it might meet marketing and financial quarterly goals, but long term success is always better when it is a good product, and for it to be a good product the people that make products and create value should determine or at least have some say when it is complete and ready.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by kelnos on Friday November 08, 2019 @03:37PM (#21486470)
"Perfect" is of course unattainable, but I'd at least hope that engineering could be able to push for -- and be granted -- the time to build a system that meets what should be some pretty obvious safety requirements around redundancy and failure detection.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by zbrozek on Friday November 08, 2019 @10:38AM (#21483147)
"Dual-dual" is a thing in this space. There are safety-intent microprocessors which have a couple of cores running a couple of cycles out of sync with a delay and comparison on registers. If they disagree after that delay, the processor concludes it's insane and you can take action based on that hardware-level signal.

It's not my favorite solution for every application, but it's a good tool for a handful of situations. And it's a good way to avoid having to think about picking who to trust with only two sockets. There are other approaches, of course. For example, if your system can afford more than a couple clock cycles of latency (e.g., if you don't believe you can crash a plane with a couple milliseconds of control error) you can do the same thing with well-designed software and a bit of extra hardware. It gives you more freedom since there's more hardware to buy, at the expense of engineering and test effort to prove that your solution does what you want.

Doing this over large physical distances also adds some challenges. There are debates around dedicated signal wires direct between control centers vs data networks. Every situation is a little different, so there aren't really easy answers that you can swoop in and prescribe as best-practices. In general I find that people are more-skeptical of packet-switched solutions (e.g., Ethernet) than I would be. But most folks are also better at reasoning about the failure modes there than they are at thinking about the lower-level issues that Ethernet solves for you. And of course there are tons of different signaling systems - not just Ethernet - to consider.

Consider just a few different possible applications and the constraints that they impose. A car has less severe failure modes (many crashes are nonfatal) but less freedom in time. Your trajectory is simply more cluttered. A passenger plane has dramatically more energy, so failures are typically more severe, but you have a lot more time to deal with them because your trajectory is empty (much of the time, anyway). And if you're sane, you've designed it to be passively aerodynamically stable because twitchy maneuverability isn't critical when you're not a fighter jet. A rocket is a different animal altogether. Those are often rigged with self-destruct explosives, and you don't really get much time to change your mind if you erroneously choose to trigger them. Plus they're typically less stable (performance is more dear, gravity isn't helping you, and purely aerodynamic solutions don't take you outside the atmosphere) and are thus more dependent on their control systems.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by MFLoon on Friday November 08, 2019 @02:00PM (#21485501)
I'm confused as to what Boeing is changing then - your description of 'dual-dual' systems sounds like what is happening with the master/slave failover system already in place. It sounds like what they're changing is to have the two systems cross-check each other in real time, rather than have the in-band delayed core be the sanity check for each system, which seems like a big difference because it means that there are now disagreement scenarios whereas an in-band sanity check always is just checking one system against one set of invariants and thus can't disagree with it's own observations. Or am I misunderstanding?
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by ethbro on Friday November 08, 2019 @02:17PM (#21485702)
You're forgetting there's a third system: the pilot.

If the majority of computer aids turn off, the plane doesn't simply drop out of the sky. It reverts back to a more complicated, but human-controllable, vehicle.

From the article (and who knows how much Bloomberg understood) the active-active sounds more like recognizing, not recovering from, errors.

Which isn't the worst idea. Pilots being, well, pilots.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by joe_the_user on Friday November 08, 2019 @03:56PM (#21486645)
I think the problem is that the 737 Max is the product of a long line of planes engineered to fly the original 737 but now flying very differently due to the massively increased engine size, which supposedly creates a much more unstable aircraft.

So you're asking the pilot to, in the midst of some other failure, to start flying a weird plane they ever flown before. You could solve that problem by training all your pilots in this BUT this solution is very expensive, so expensive the airlines might just demand their money back instead. And that wouldn't be a solution for Boeing.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by ethbro on Friday November 08, 2019 @04:11PM (#21486811)
All true. But from where we are today, that seems like a better solution from a safety perspective.

And now, the FAA (or EASA) might just mandate that anyway.

I wouldn't be surprised if CAAC refuses to ever recertify the 737 MAX, especially if a trade deal isn't struck.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by zaroth on Friday November 08, 2019 @09:30AM (#21482436)
As a tangent, there have been times when HN is ahead of the curve in understanding where something is headed before the investing public.

The benchmarks and excitement around AMDs offerings are a great example. When you see something that presents a paradigm shift, often the market will not have fully digested it.

AMD is one of those cases, and I believe BA is as well. Their stock is essentially unchanged from where it was before the Lion Air crash in October 2018. This company is worth $200B and we’re talking about 5,000 units of a $100mm plane (that’s $500B in revenue right? I’m not adding a zero there?)

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by Xixi on Friday November 08, 2019 @09:58AM (#21482756)
$100mm is the list price. Actual prices are decided by the market. I've read about $50mm for the 737-8, the baseline 737MAX model. That was of course before the MCAS disaster.

My layman understanding is that actual prices would be around 40% off list price for models that sell really well, with little competition (for instance A321). Much steeper discount above 60% for models that have a hard time selling (A330NEO) or for end of line to close a production gap (777-300ER is very cheap to buy while Boeing is working on the 777-9).

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by chkaloon on Friday November 08, 2019 @10:17AM (#21482926)
That, or this article missed the boat somehow. I find it hard to believe that a total redesign is happening and that they expect to be done by the end of the year. Either this article is wrong, or something is terribly wrong at Boeing.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by dboreham on Friday November 08, 2019 @10:24AM (#21483006)
I'm a huge MAX skeptic, but this article just has to be wrong. Someone who had dropped out of a CS degree would know not to do what the article says they're doing.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by SQueeeeeL on Friday November 08, 2019 @12:04PM (#21484202)
You would be surprised, if managament is broken to it's core, the situation is "Do this unethical thing or we'll find someone who will!"

Computer Scientists aren't usually taught engineering ethics and definitely get fired all the time for arbitrary reasons.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by uxp100 on Friday November 08, 2019 @12:18PM (#21484363)
Is that true that CS students don't take an ethics course? I have a 10 year old SE degree and I took 3, only 1 of which was engineering specific. (I took a philosophy ethics course, which really isn't that relevant to issues like these anyway, but also a course that was specific to ethical issues in computing.)

OTOH, I'm skeptical those courses change behavior, and in this particular case others in comments are saying this is a common architecture in the space, so how could anyone solve this ethical conundrum if getting to the base level of whether this is an ethical conundrum to begin with is difficult?

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by SQueeeeeL on Friday November 08, 2019 @12:26PM (#21484463)
In my opinion courses wouldn't change behavior, modern computer science is designed to infantalize coders and make them interchangeable. Think of all the effort startups put into marketing themselves as "COOL TO WORK FOR"/"WE HAVE BEER ON FRIDAYS". These reinforce the idea that you are small and need management to tell you exactly what to do everyday, because you're a dumb idiot who spends all day watching anime and playing video games.

In that environment, serious ethical concerns could easily be brushed aside as "coders just being coders". When Google got found out for making drones, the employees of a company staged a 5 minute walkout.... that was it, I'm sure some people quit, but not enough to make any impact

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by thfuran on Friday November 08, 2019 @02:17PM (#21485710)
Given

>marketing themselves as "COOL TO WORK FOR"/"WE HAVE BEER ON FRIDAYS",

"the idea that you are small and need management to tell you exactly what to do everyday, because you're a dumb idiot who spends all day watching anime and playing video games" does not follow.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by SQueeeeeL on Saturday November 09, 2019 @02:32AM (#21490183)
They make efforts to cast the coders as young people who can be swayed by marketing gimmicks instead of real workplace concerns like benefits, livability, consistent work/life balance. That culture permiates the entire engineering culture
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by bradstewart on Friday November 08, 2019 @01:08PM (#21484981)
I took one course that briefly touched on engineering ethics in my ECE program. Nothing even remotely resembling the ethics courses taken by the Mechanical/Civil/Structural guys.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by ashleyn on Saturday November 09, 2019 @09:38AM (#21491772)
I had one ethics course. Class of 2013 in a prominent Pennsylvania university.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by MFLoon on Friday November 08, 2019 @01:45PM (#21485373)
SE = structural engineering? That's cool, I'm glad your discipline is receiving some ethical training. I've never heard of any CS program including any sort of engineering or general ethics courses in their curricula.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by tofof on Friday November 08, 2019 @03:24PM (#21486330)
Every single CS baccalaureate degree from the University of Illinois has required from at least 2005[1] a mandatory engineering ethics course (CS 210).

Note that the UofI has always been in the top ten nationally for CS; its 2020 rank is #6 (behind MIT, CMU, UCBerkeley, Georgia Inst. of Tech, and Stanford).

1: 2005: https://web.archive.org/web/20050321215749/http://courses.ui...

2: 2019: https://cs.illinois.edu/academics/undergraduate/degree-progr...

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by uxp100 on Friday November 08, 2019 @04:10PM (#21486805)
Yeah, as others said, by SE I meant software engineering. Lots of overlapping acronyms, I usually don't use them, but in this context I thought I'd get away with it.

I liked my ethics course in the philosophy department, but that's more about J.S. Mill, and Kant and the rest. Engineering ethics was a course any Eng. degree had to take, interesting, but hard to apply also. You go through case studies and look at them, but to have any productive class discussion they kinda have to be broken down to be at least somewhat clear and concise.

I don't feel like getting into here, but I feel like even this Boeing issue, which is pretty clear, is less clear than any case study in a textbook.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by TickleSteve on Friday November 08, 2019 @02:51PM (#21486033)
My CS degree certainly did from University of Manchester.

Its also my understanding that most do.

BTW: SE typically means Software Engineering in this context.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by chkaloon on Friday November 08, 2019 @10:43AM (#21483219)
I wonder if someone talked to this reporter about a long-term redesign they're doing, and the reporter conflated that with the short term patches they're doing to get the plane flying again. It just doesn't make sense otherwise.

And, 100 scenarios instead of 10? I sure as heck there's more than that on an architecture redesign!

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by EricE on Friday November 08, 2019 @03:35PM (#21486448)
No kidding! How sad is it the entire time I was reading this I kept looking back at "Bloomberg" and was thinking to their other reporting like the server spying that they never did explain.

The National Enquirer may be more reliable than Bloomberg at this point.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by Merrill on Friday November 08, 2019 @10:27AM (#21483032)
>Landing contracts for key systems on the Max cemented a relationship with Boeing that Collins had long cultivated, in part by creating what its chief executive in 2002 called an “enormous sea change in a very long legacy business." For years, the company had been “a technology leader, an innovator,” Clayton Jones, then the chief executive, told Fortune magazine. “Unfortunately, along the way they forgot to hone their financial skills.” In 1998, Boeing executives summoned Mr. Jones to Seattle, he later recalled in a speech, and made clear that, to get more of Boeing’s business, Collins would have to cut prices dramatically. In response, Collins introduced what it called “lean electronics,” its take on a belt-tightening philosophy popularized by Toyota.

>Collins reorganized business units and retrained managers, with an eye toward efficiency and speed. It pushed its suppliers to do the same, and established partnerships with companies such as HCL Technologies, which provides outsourced, lower-cost engineering services from India.

>Along the way, Collins unseated Honeywell as the provider of flight control computers on a predecessor of the Max, the 737 NG, and supplied numerous systems for the Boeing 787, which went into operation in 2011.

>When Collins secured the contract for the Max displays in 2012, it credited the belt-tightening. “There were a lot of cost-saving measures — a lot of tough decisions — that had to be made,” one manager said at the time in a company publication.

https://www.nytimes.com/2019/11/07/business/boeing-737-max-c...

If there was a problem with the master-slave operation on the MAX, not necessarily connected with MCAS, might not the same problem exist on the 737 NGs?

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by dk- on Friday November 08, 2019 @10:34AM (#21483096)
not likely the NG didn't add new systems. not saying other cost-cutting won't be exposed but nothing as egregious as MCAS.

the MAX added new engines, which required forward placement on the wings, which changed COG, which necessitated MCAS.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by Merrill on Friday November 08, 2019 @10:46AM (#21483251)
They changed flight control computers from Honeywell to Collins when going from the 737 to the 737 NG. Seems like a new system.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by DuskStar on Friday November 08, 2019 @11:34AM (#21483822)
Right, but the recent change was from the 737 NG to the 737 MAX.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by dhimes on Friday November 08, 2019 @09:25AM (#21482397)
I hope they aren't expecting people to fly on these planes.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by xvf22 on Friday November 08, 2019 @09:40AM (#21482533)
A split brain in a plane is a bit of a nightmare. I'm not sure how they can work around it with just two computers.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by imglorp on Friday November 08, 2019 @09:48AM (#21482636)
Big "disagree" alarm for the pilots and let them decide to keep either computer or go direct. Airbus has the concept of flight control laws, surfaced and changeable by the wet backups.

http://www.airbusdriver.net/airbus_fltlaws.htm

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by 0xffff2 on Friday November 08, 2019 @11:00AM (#21483460)
>go direct

That's not really a thing in modern airliners. The computer is still heavily involved even when hand-flying the plane.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by DuskStar on Friday November 08, 2019 @11:37AM (#21483865)
"Thankfully", the 737 series isn't a modern airliner, and isn't fly-by-wire. All control inputs are directly linked to the relevant outputs, though sometimes with power assistance.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by imglorp on Friday November 08, 2019 @01:16PM (#21485060)
It is on the Airbus, one of the control laws, see link above. It's still FBW though, so one might argue that "mechanical backup" is really what you want.

As for "modern", the A320 had control laws in 1984 and the 777 in 1994.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by TomVDB on Friday November 08, 2019 @10:28AM (#21483045)
And Boeing will make that disagree alarm an optional extra?
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by cmurf on Friday November 08, 2019 @10:40AM (#21483183)
Am optical extra that isn't actually properly functioning as intended in numerous delivered products? Almost as if it wasn't sanity tested in one production unit, let alone some random sample of units?
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by cmurf on Friday November 08, 2019 @02:46PM (#21485982)
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by linuxftw on Friday November 08, 2019 @10:21AM (#21482969)
Which computer calculates that there's a disagreement, and how can we ensure it's not indicating a false negative?
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by imglorp on Friday November 08, 2019 @10:22AM (#21482989)
Either. The alarm must be an OR.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by linuxftw on Friday November 08, 2019 @10:36AM (#21483127)
Then, IMO, it should be two independent indicator lights, not 1.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by dk- on Friday November 08, 2019 @10:36AM (#21483122)
my CS201 class 19 years ago taught me this. education just isn't as good these days..
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by beamatronic on Friday November 08, 2019 @07:00PM (#21488320)
I didn’t feel this way at first, but now I feel like I will never fly on this plane or allow my family members to fly on this plane.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by nobodyandproud on Friday November 08, 2019 @02:59PM (#21486107)
tl;dr : The Kruger-Dunning effect is most visible in the upper-management circles, where politics and charismatic talents are necessary skills.

Former engineering powerhouses like HP, Boeing, and IBM slowly get hollowed-out into incompetence when it’s assumed that Engineering and technical skills don’t require a considerable depth of knowledge and experience.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by bdavis__ on Saturday November 09, 2019 @06:01AM (#21490794)
profound!

could it also be the normal generational rotation of people through the organization?

Everyone grew at a tremendous rate since the 1960's. Many, many more engineers. And then they retired.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by nobodyandproud on Saturday November 09, 2019 @10:31AM (#21492056)
I don’t think it’s generational. It’s just unlearned lessons of history.

The most effective leadership for a technical company is leadership with actual technical creds to back it up.

The technical side is a must because you need someone who actually understands the business and the innovations that must takes place. But you also need someone who can provide the people-leadership skills and the business wheeling&dealing.

These are all crucial skills and take up a persons time, so it’s not uncommon to see this come in the form of a duo (Apple and Steves, or Microsoft with Bill+Steve).

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by tus88 on Friday November 08, 2019 @02:35PM (#21485873)
> bad software

The software was fine (or at least not to blame). Unstable aeronautic design and lack of redundant sensors was the real problem.

Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by carlmr on Friday November 08, 2019 @10:27PM (#21489374)
Looking at the current article the software is also low quality.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by heisenbit on Friday November 08, 2019 @02:12PM (#21485652)
If what we read is true this is then the second time that Boeing is trying to fix hardware shortcoming with software in a hair raising manner.
Re:Delays in Boeing Max Return Began with Near-Crash in Simu... by speg on Friday November 08, 2019 @07:28PM (#21488498)
What is TFA?