News for hackers. Stuff that matters.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by crorella on Friday June 28, 2019 @06:17PM (#20309227)
Money saved by firing experienced engs and outsourcing the development << money lost because they can't sell planes.

Why is that so hard to understand? Massive fail at risk management maybe?

Re:Boeing's 737 Max software outsourced to lower-paid engineers by meowface on Friday June 28, 2019 @06:26PM (#20309302)
For one, in part because many managers don't really understand software or software projects and think it's mostly just people * man-hours. "Through this cost-saving measure, we've tripled the amount of people working on the project and expect to reduce the amount of time it'll take to complete by two-thirds without incurring any additional costs." In their minds, it's an easy win.

And they were probably regularly reassured by various internal and external entities that the code was high-quality and thoroughly tested and very safe. Obviously they're not going to outsource if they think it's likely to cause a safety hazard; they just didn't place the odds of this outcome anywhere near what the actual odds ended up being.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by nobodyandproud on Friday June 28, 2019 @09:09PM (#20310144)
Which means they have no business managing. It’s to tiresome to repeat because it should be obvious , but engineering is not a commodity.

The end product may be, but not the design problem process nor the manufacturing process.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by xiphias2 on Saturday June 29, 2019 @09:32AM (#20312467)
For me the solution is quite easy: I don't invest in companies where I don't see technical excellence happening.

The world is changing fast enough that old style management won't get investment after some time.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by weaksauce on Friday June 28, 2019 @08:36PM (#20310024)
someone needs to throw the mythical man month at these managers.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by durnygbur on Saturday June 29, 2019 @03:18AM (#20311264)
There would be no englightment happening.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by RachelF on Friday June 28, 2019 @11:21PM (#20310630)
Most managers only think short-term. In the short term it saved money. In the long term is cost Boeing their reputation.

But in the long term the MBA manager has been likely been promoted and is working elsewhere, killing another company, while his or her CV boasts of "reducing inefficiencies".

Re:Boeing's 737 Max software outsourced to lower-paid engineers by ensiferum on Tuesday July 02, 2019 @03:46AM (#20333046)
As long as they can get a RSU package and get a promotion it's s win.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by ganeshkrishnan on Friday June 28, 2019 @09:32PM (#20310241)
The software did exactly as it's supposed to do: If the AOA sensors tells you the plane is tilting too high up, pitch the nose down a little.

The main issue was that there was only one AOA sensor connected in standard packages and the hardware for it failed.

The pitot tubes failed for the doomed Air France incident which caused the software to read incorrectly and stall.

For what it's worth AirBus has a huge engineering department in India: https://www.airbus.com/careers/search-and-apply/opportunitie...

Articles like this are just drummed up to stir negative emotions about "others" (with "struggling to understand English" and "unable to code fizzbuzz") as is evidenced in the rest of the comments below.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by dTal on Saturday June 29, 2019 @08:54AM (#20312308)
This comment is glib and misinformed.

Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that. And your description of the MCAS specification is vastly over-simplified - it is not an autopilot and its job is to gradually change the behaviour of the plane in different flight modes (which it must detect), not to directly fly the plane as you suggest (and even two AOA sensors would be insufficient for the type of functionality you describe). The behaviour of the MCAS is absolutely in question, in particular the way it kept "re-adding" the correction, with cumulative effect, ending up fully pitched down.

Secondly, if by "the doomed Air France incident" you mean AF447, your description of it is flat-out wrong. The pilot, not software, stalled and crashed the plane by manually holding the stick back for 15 minutes. The pitot tubes briefly iced over (they didn't "fail") which is not an uncommon occurrence, and the only consequence of this was that the software guardrails that would have prevented the pilot from crashing the plane were removed. All instrumentation systems were fully operational at the time of the crash.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by sokoloff on Saturday June 29, 2019 @12:38PM (#20313466)
Agree with everything you wrote, with the possible exception of referring to the stick-holder in the AF447 crash as a "pilot".
Re:Boeing's 737 Max software outsourced to lower-paid engineers by mannykannot on Tuesday July 02, 2019 @12:19PM (#20336595)
> Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that.

This belongs in the true-but-trite category. The decision to depend on a single, unverified AofA input was not a programming error, and neither was the increase in power in the second version. These decisions were made by Boeing, and endorsed by the FAA (to the extent that Boeing informed the FAA of them.)

Re:Boeing's 737 Max software outsourced to lower-paid engineers by ganeshkrishnan on Saturday June 29, 2019 @09:23AM (#20312418)
It kept re-adding because the flap of the aoa sensor was stuck in the "aircraft is pitching up" position.

Boeing management did not add a second sensor to some planes and also pilots were not aware of how to switch off mcas.

If you worked in large scale project you would know that most devs are assigned components to work on and usually have no idea what other components are or how they will interact.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by gotdevops on Saturday June 29, 2019 @10:14AM (#20312696)
This is where testing comes in and from the fact packet it appears that the testing was also outsourced to another Indian firm?
Re:Boeing's 737 Max software outsourced to lower-paid engineers by AlexCoventry on Saturday June 29, 2019 @12:31AM (#20310820)
> Articles like this are just drummed up to stir negative emotions about "others"

The negativity seems more targeted at the leaders who arranged this than at the developers. Both the examples you cite are from the same comment, and even that comment is more about resentment towards management corruption than about the incompetence of the developers hired as a result of the corruption.

> The software did exactly as it's supposed to do

I gather new issues with the software have come to light since the accidents, as a result of the heightened scrutiny, and they seem likely to keep the planes grounded for longer than Boeing hoped.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by danielg6 on Saturday June 29, 2019 @02:09AM (#20311036)
See the post below yours lol.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by AlexCoventry on Saturday June 29, 2019 @01:04PM (#20313603)
Not sure which one you were referring to. The ordering of comments changes over time. Could you post a URL or identifying substring?
Re:Boeing's 737 Max software outsourced to lower-paid engineers by danielg6 on Saturday June 29, 2019 @07:34PM (#20315496)
That’s the joke... You can scroll up or down and find such a comment.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by gotdevops on Saturday June 29, 2019 @12:05AM (#20310755)
You sound suspiciously like every other outsourcer I’ve worked with who breaks something critical. A million reasons why it wasn’t their fault. They did exactly what they were told right and didn’t take one second to validate the assumptions surrounding the systematic nature of the solution. The goal, the mission, the end state solution. That was another departments concern right. That was QA’s concern, that was designs concern, that was the testing center of excellence’s concern. Blame them. The perfect circle of siloed fiefdoms and no feedback blame. The problem is that lives were at stake here on this one and the entire house of cards is coming crashing down if this is determined to be true. There will be public outcry and no matter how many excuses you have the fact is families and lives have been ruined FOREVER, NOTHING ON THIS EARTH CAN REPLACE THEM and something must be done.

https://qualitysafety.bmj.com/content/13/suppl_2/ii22

Re:Boeing's 737 Max software outsourced to lower-paid engineers by danielg6 on Saturday June 29, 2019 @02:06AM (#20311031)
I do think it’s weird that the parent post started out by saying the sensors were designed to spec, but is he wrong? The article even said as much. (Not MCAS though.) I didn’t read it as an excuse.

It seems like you’re saying that outsourced devs should have as much accountability as home-based devs or the entity Boeing itself. I don’t agree with this. The salary does not reflect that. The work that’s assigned does not reflect that. Anecdotally working with outsourced devs and based from the comments here, no one honestly thinks that.

It seems like you were triggered by the “excuses” part of the post. But what about the rest of it? He made a valid point (which I think you proved) about people using the article to discriminate.

I guess I’m just wondering why you’re making generalizations. I’ve worked with many devs (who weren’t outsourced) who blame everyone else but themselves. I learned to use ‘git blame’ very early on.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by gotdevops on Saturday June 29, 2019 @10:55AM (#20312928)
It’s a very interesting point. I’ve worked with some very skilled devs from HCL. I’ve also seen them turn over to other firms because of salary and other firm related issues. I don’t think that piece has anything or very little to do with the fact that they are Indian. However the organizational culture is definitely flawed and it does appear to be more prevalent in large fixed bid outsourcing arrangements. I’ve witnessed it first hand being brought in to resolve issues created by the outsourcer on projects as large as 1/2bn. They difference between those projects is the mission. Those project were critical on a financial basis. This was and aircraft with peoples life’s on the line. No one dev or otherwise can say they didn’t know this code was going on an airplane. No matter their salary.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by danielg6 on Saturday June 29, 2019 @12:09PM (#20313325)
Thanks. I thought your first post was very emotional and didn’t align with the truth of the situation, which is why I felt the need to reply.

You’re right in that the primary focus of the article was not the outsourced devs being Indian. The issue comes from how people respond to such articles, no matter the article’s intention.

I’m not saying that outsourced devs shouldn’t be held accountable for any faults. The point is that there are different degrees of accountability for all parties involved.

The article only chose to focus on one of these groups, which is fine. What’s not fine is HN comments that seem to not realize this.

No matter the salary or experience or contract, outsourced devs/companies shouldn’t be in a position of power to break critical systems on an airplane. (I’m not saying this is what happened with the Max.)

Re:Boeing's 737 Max software outsourced to lower-paid engineers by dang on Saturday June 29, 2019 @03:25AM (#20311284)
Could you please not cross into personal attack? and also please not post in the flamewar style? These are things we're trying to avoid on HN. The site guidelines explain, and if you'd read them and stick to the rules, we'd be grateful: https://news.ycombinator.com/newsguidelines.html.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by mannykannot on Tuesday July 02, 2019 @12:31PM (#20336727)
In your world, Boeing designed (and the FAA endorsed) MCAS to use only one sensor and made it dangerously powerful, yet the crashes are the coders' fault for being the ones who did not anticipate the problems this would cause.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by TarlanT on Saturday June 29, 2019 @09:17AM (#20312395)
Experienced software engineer would NEVER code something like that. Both physically and logically it makes no sense to get same AOA reading after many iterations of trimming. That clearly means something is wrong. And probably the sanity of data being fed is in question. It’s the dumbest mistake in critical system I’ve ever heard. Even in non critical software these types of logical mistakes are rare.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by mannykannot on Tuesday July 02, 2019 @01:09PM (#20337137)
AofA is a function of a number of factors, such as control inputs -- data not available to MCAS.

It would be disturbing if the coders on aircraft control systems were modifying the specifications on the fly, and on the basis of misconceptions like that displayed here.

Responsibility for the dangerous capabilities of MCAS lies with Boeing, which chose to fix an aerodynamic problem this way, and the FAA, which endorsed it.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by nobodyandproud on Saturday June 29, 2019 @02:00AM (#20311008)
The problem is that developers also need to understand what they are coding, and fight back when they notice that the design is fundamentally flawed.

In this case, the lack of redundancy should have been a red-flagged.

This is why domain knowledge is at least as important as coding skills.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by durnygbur on Saturday June 29, 2019 @03:21AM (#20311275)
In highly process-based environments, which outsourcing centers are, fighting back gets you harrased by the zealot managers out of the workplace.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by ganeshkrishnan on Saturday June 29, 2019 @09:46AM (#20312532)
There were multiple redundancy built in. Multiple sensors, voting mechanism, manual override. Many of which meant re-training the pilots and Boeing wanted to avoid it.

I think southwest airlines had multiple sensors

Re:Boeing's 737 Max software outsourced to lower-paid engineers by nobodyandproud on Sunday June 30, 2019 @04:19AM (#20317039)
The 737 Max aircrafts that crashed relied on a single AOA sensor.

A second AOA sensor is an option.

Which means code was written to account for both configurations.

Many, many things must happen and change to avoid allowing management to make decisions unchecked.

Because this is a case of bargain-bin engineering and tunnel-visioned management ruining a once-reputable company.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by pxue on Friday June 28, 2019 @09:51PM (#20310316)
Software was there when we went to the moon and back to ensure hardware failures are handled and communicated. Why can't it do the same for these planes?
Re:Boeing's 737 Max software outsourced to lower-paid engineers by ganeshkrishnan on Friday June 28, 2019 @09:56PM (#20310331)
https://en.wikipedia.org/wiki/List_of_spaceflight-related_ac...

Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs

Re:Boeing's 737 Max software outsourced to lower-paid engineers by gotdevops on Saturday June 29, 2019 @12:08AM (#20310764)
It’s the hardware people’s fault right. Oh no it’s managements fault. It’s everyone else’s fault. I see a pattern forming here.

https://qualitysafety.bmj.com/content/13/suppl_2/ii22

Re:Boeing's 737 Max software outsourced to lower-paid engineers by rossdavidh on Friday June 28, 2019 @08:03PM (#20309877)
I have seen this kind of problem before (never in a case where lives were on the line), and I called it "pathological optimism". If you are in a corporate culture of the sort that lionizes "can-do" thinking, then saying "that won't work" is going to look defeatist.

I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. But more likely, it's just that pathological optimism works, really well, right up until that moment when it doesn't. Taking risks in order to lower costs makes the bottom line better, which seems to call for more of the same, right up until the moment when the risk explodes in your face.

It is not unlike the attitude towards software security, in many industries. People harping about security risks seems like just all cost and no benefit, right up until the point where you have a big problem.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by CydeWeys on Friday June 28, 2019 @08:08PM (#20309900)
> I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy.

If so, they're learning the wrong lessons. SV unicorns become valuable because of explosive growth, not because they underpay their engineers. Just the opposite; a lot of engineers at SV startups are paid quite handsomely (especially the early ones at successful unicorns). SV startups don't win on cost-cutting.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by gotdevops on Saturday June 29, 2019 @11:14AM (#20313021)
I think you are right on with this. Most startups also leveraging an organizational culture that goes against silos or fiefdoms. In the startup culture I’ve been a part of fast feedback and early identification of flaws in the process is a cornerstone of the development process. Troubleshooting issues in production is the worse case scenario. In this case production is literally a flying airplane.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by pjc50 on Friday June 28, 2019 @06:31PM (#20309326)
The risk management is being TBTF. In a nationalist era, is the US going to buy planes from Airbus, or are they going to bail out Boeing?
Re:Boeing's 737 Max software outsourced to lower-paid engineers by broahmed on Friday June 28, 2019 @07:54PM (#20309822)
For others wondering: TBTF = too big to fail
Re:Boeing's 737 Max software outsourced to lower-paid engineers by cronix on Friday June 28, 2019 @06:20PM (#20309250)
Can't sell planes, or fly the ones that exist, and pay out huge untold sums in death lawsuits.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by crorella on Friday June 28, 2019 @06:21PM (#20309261)
Yeah, that too, after the fact seems so stupidly obvious.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by tarsinge on Saturday June 29, 2019 @10:11AM (#20312677)
Unfortunately it seems other critical industries are affected, reminded me of the EPR nuclear reactor fiasco https://en.wikipedia.org/wiki/EPR_(nuclear_reactor)#Olkiluot...

Olkiluoto (Finland)

> In May 2006, construction delays of about one year were announced, following quality control problems across the construction. In part the delays were due to the lack of oversight of subcontractors inexperienced in nuclear construction.

> At the end of June 2007, it was reported that Säteilyturvakeskus (STUK), the Finnish Radiation and Nuclear Safety Authority, had found a number of safety-related design and manufacturing 'deficiencies'.

> Initial cost estimates were about €3.7 billion, but the project has since seen several severe cost increments and delays, with latest cost estimates (from 2012) of more than €8 billion.

Flamanville (France)

> In April 2008, the French nuclear safety agency (Autorité de sûreté nucléaire, ASN) reported that a quarter of the welds inspected in the secondary containment steel liner are not in accordance with norms, and that cracks have been found in the concrete base.

> In April 2015, Areva informed the French nuclear regulator ASN that anomalies had been detected in the reactor vessel steel, causing "lower than expected mechanical toughness values". Further tests are underway.[64] In July 2015 The Daily Telegraph reported that Areva had been aware of this problem since 2006.[65] In June 2015, multiple faults in cooling system safety valves were discovered by ASN.

Re:Boeing's 737 Max software outsourced to lower-paid engineers by bsder on Friday June 28, 2019 @06:55PM (#20309473)
> Why is that so hard to understand? Massive fail at risk management maybe?

Managerial bonus or promotion for saving money this quarter >> money lost more quarters out than I'll be around

Re:Boeing's 737 Max software outsourced to lower-paid engineers by gscott on Friday June 28, 2019 @08:57PM (#20310083)
This is really it. Board of directors set goals, say reducing costs, they offer a big bonus, you meet the goal, get the big bonus. Problems become someone else's proble. You buy a big car, house, go to fancy non-profit's $500 dinners.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by gotdevops on Saturday June 29, 2019 @11:27AM (#20313092)
This is definitely one major issue. At the top. The challenge at the top is that these outsourcing contracts are for many millions of dollars. They are often written in such a way that the parent cannot dictate how the outsourcing firm conducts business. The parent fills out a ticket and the outsourcer assigns the ticket to a resource and marks the ticket complete. The issue here is that this is not a bug related to email or Login credentials. This is an airplane with folks relying on it for safety of flight.
Re:Boeing's 737 Max software outsourced to lower-paid engineers by dongping on Friday June 28, 2019 @06:31PM (#20309327)
The managers who saved money would have been promoted well before any accident happens.