Consider something like the 737 MAX debacle – did the programmers writing the MCAS code actually have enough aviation domain knowledge and understanding of where the component fit in the overall system to realise it was a threat to people's lives?
I don't know, but my guess is the most likely answer is "No".
The MCAS is an optional component that reduces certification and training costs. It is definitively possible to fly the plane without accidents even with a disabled MCAS. So why can't the MCAS be turned off automatically when sensors fail? Because that changes the classification of the plane and therefore requires pilots to be certified for a new machine and receive new flight training for both MCAS and no MCAS modes.
If the software engineer was under a hippocratic oath then he would have to refuse to build the MCAS entirely but not because the idea of an MCAS is inherently unethical, no, he would have to refuse because the company he works at wants to use the MCAS for a non ethical purpose (namely operate and hide the existence of MCAS even when it is unsafe to do so).
This is basically a reverse audit but the software engineer has no authority conduct such an audit and even if he was allowed to, the business has no obligation to give him the necessary information to determine whether the MCAS will be used unethically.
You think a programmer, handed a spec and asked to implement it, can be expected to know that their employer (or the employer's customer) wants to use it for a "non ethical purpose"?
Again, I can't know for sure, but I doubt the programmers who wrote MCAS (who most likely didn't even work for Boeing, but rather some subcontractor) actually knew, or could have known, how the code fit into Boeing's larger purposes
Most types of development in these large companies is so compartmentalized that it's next to impossible to see the whole structure from a software engineers prospective. You need to be at a management level to understand how most of the pieces really come together, which is the only place where one of these "oaths" might make an influence. At that point, however, the selection is so goal oriented, I have a doubt as to whether or not people would take that oath.
Of course, there’s all kinds of pressures that make these fall through the cracks. I vaguely remember an article stating some of these documents in the case of MCAS were not up to date
The HA listed MCAS as "hazardous" rather than "catastrophic". Meaning those in charge of that process document did not realize MCAS had the ability to down the airplane. I know it's tempting to arm-chair quarterback this, but let's assume they should have realized this hazard.
To your point, maybe the programmer doesn't have the systems knowledge to make those calls, but the process is predicated on somebody having both the technical acumen and the responsibility) for those decisions. This process broke down though.
No imtringued does not.
imtringued wrote:
> This is basically a reverse audit but the software engineer has no authority conduct such an audit and even if he was allowed to, the business has no obligation to give him the necessary information to determine whether the MCAS will be used unethically.
imtringued is saying that it would be impossible for a software engineer to determine whether what they were asked to do was ethical or not.
One, it's not going to be clear from the request that the MCAS would want to be used in unethical ways.
Since the Hippocratic oath is the argument here, how many software developers want to work in a system closer to physicians? A national cartel controlling membership and licensure - tough luck if you want to hire more developers because there's an artificially limited supply. Mandatory academic training - goodbye self-taught developers. Follow-on training with pay 1/5th or less of your attending physicians - I know residents in specialties where attendings are paid $500k a year to start, and they are making $60k a year. Brutal shift work - residents work 70-80 hours a week easily. Toxic leadership - I've heard horror stories of residents being forced to lie on ACGME forms regarding their hours under penalty of being outright fired from their residency slot, which would make it nearly impossible to get a job as a physician (mainly because you'd have to apply to a different residency program and explain your termination).
I know they're not suggesting bringing the entire medical education & training structure over to tech workers, and everyone here likes to think that they're brilliant and changing lives every day but most of us are just throwing shitcode JS into a computer for 3-4 hours a day for an ad tech company and not much more. The comparison falls apart pretty quickly.
This isn’t exactly true. There are mitigations (both software and non-software) that are expected to be done depending on hazard analysis. One of the items discovered is Boeing mischaracterized the MCAS hazard (it should have had a “catastrophic” hazard class). In addition, they didn’t appear to follow their own process for dual inputs required even for the lower severity class assigned. The “optional” part of MCAS was the secondary sensor reading into the software
No. The MCAS was a "necessary" component for pitch stability - without it, a 737 MAX in a pitch-up attitude would, in the absence of correcting inputs, pitch up further and further until it stalled. Without it, the airframe is uncertifiable, full stop.
I'm certain that's not correct, everything I've read on it has said MCAS was specifically a software modifier put in place to allow the plane to respond substantially the same as a regular 737 without the larger engines, in order to avoid having to have additional training for all 737 pilots worldwide.
Most aircraft, in a "pitch up attitude" will increase their angle of attack as thrust is applied. The issue was that the MAX would do so in a more radical way than the regular 737 did, and so the software was put in place to limit that so it flew like a regular 737 as far as the pilots could see.
Conceptually, MCAS wasn't a bad idea. The execution and using it as a replacement for training and not informing pilots of the flight characteristics changes between the models was stupid.
Although to be fair my summary wasn't entirely accurate - it wasn't that a MAX was outright dynamically unstable with no control input, as I described, but rather not sufficiently stable as to cause a monotonic increase in stick force as AOA increases, which can cause the combined system of pilot + flight dynamics to be unstable since the pilot relies on stick force as an indicator.
> Most aircraft, in a "pitch up attitude" will increase their angle of attack as thrust is applied
This is both incorrect and irrelevant. Most aircraft will climb when power is applied, but will not change their AOA unless the thrust axis is off-center. To a first approximation, power controls climb rate, and stick input changes AOA. Change in behavior under different power settings has little to do with the problem with the MAX. The problem with the MAX is that at high angles of attack - i.e., when the stick is held back, causing the air to meet the wing (and the engines) at a steeper angle - the engines, which are flung forward, start producing lift of their own and produce a pitch-up moment. This means that the further the pilot pulls the stick back, the less hard they have to pull. This is a dangerous inversion that increases the control order of the system, as it breaks the usual assumption that a given stick force will result in a given AOA, more or less.
The original 737 does exactly the same thing the Max does with respect to producing a pitch-up moment -- as does nearly every other aircraft. It's just not nearly as pronounced as the Max is.
>The Boeing 737 MAX MCAS system is there ONLY to meet the FAA longitudinal stability requirements as specified in FAR Section 25.173, and in particular part (c) which mandates "stick force vs speed curve:, and also FAR Section 25.203 — "Stall characteristics".
For this to work, software engineer compensation would need to change radically.
A software developer refusing a job because it does not meet his ethical parameters is just an unemployed software engineer.
There are definitely areas where the prudent thing for a developer to do is raise a dissenting opinion, if not halting work. What seems lacking is clear industry consensus standards to back up that decision.
For the same reason, it's harder to hire some rando budget doctor because the field is gate-keeped by the requirement of a licence, and liability.
You can't magically make Engineering the same without the same conditions. Add barriers top entry that see my pay rise, or have cheap programmers with no liability.
I’ve never been on a project that requires a software product stamped by a licensed engineer. NCEES dropped the software license because there was so little demand, compared to, say, civil engineers who consider a license a rite of passage to career growth.
1) you work for the federal government
2) you work under a licensed engineer
3) you work under an industrial exemption
There’s differences depending on state. There has been a more concerted effort to remove #3 recently due to both political reasons and the technical issues in this thread. Most people performing engineering work under an industrial exemption don’t actually realize it. Again, this is different state to state. For example, in some states you can’t start a business with “engineering” in the name unless you have a certain percentage of owners/principals with an engineering license.[1]
What actually happens is conflating terms in common parlance. “Engineer” and “engineer” are not necessarily the same. For example, a computer engineer may work under an industrial exemption (due to working in a manufacturing service) while a software engineer does not. Legally, an “Engineer” claims an explicit responsibility to public safety.[2] Apropos to the headline article, there is a distinction with this difference.
[1] https://fxbinc.com/wp-content/uploads/2016/10/state-by-state...
[2] https://www.theatlantic.com/technology/archive/2015/11/progr...
The reason MCAS came about was because management wanted to try to fudge a larger engine into an outdated design created for a different purpose rather than do the engineering and certification necessary for the new requirements and to update the system.
Management wanted to save money. Of course the engineering leadership did not want to fudge something -- they wanted to do proper engineering. But the people in charge just wanted to save money, and the engineering leadership could not do anything otherwise, even they knew that just making the engine larger and compensating did not make sense from an engineering standpoint.
By the time it got to the MCAS, that was far down the line of the decision to not do proper engineering.
Blaming it on management is also irrelevant, since management merely takes the advice of engineers, and do financial/business trade offs to maximize profit. If the engineers cannot tell that MCAS system could fail this way (due to the complexity), management will not question it.
if the engineering leadership said, this is not a good idea. Management said, it saves money.
Engineering failed to convince management. Management didn't have the understanding that it was a bad idea.
It is now, no one's fault?
If management merely takes the advice of engineers ( and other who specialize in the things that they do not ), and they choose to ignore it because they do not understand the things they do not specialize in. I believe it's a reasonable to assume that management is more at fault than engineering ( I'm not sure they're is a situation here where any party is fault less )
Management likes hearing things they like, and simply don't hear things they don't like. Then act surprised about it when it becomes public.
Its very common.
By the rest of your comment, it looks as though it's an excellent comparison. How would engineers taking an oath help the situation?
This is a red herring. The oath is not needed merely for illegal work. In fact, the more common use cases will likely be legal. It's a common sentiment, but: Don't conflate ethics with legal.
> Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself.
This is not a dichotomy - there can be a spectrum. You can restrict it to those who do know what the product is used for, or at least have good guesses for them.
And while not everyone is this way, I wouldn't really want to work in a job for long if I'm not told what the code I'm writing is for. It's not even an ethical concern for me - it just makes for a boring job. Ideally I want people to tell me the problem they are solving and give me some leeway in crafting a solution. Don't come to me with a solution and ask me to implement it.
But money dude.
Also, the Hippocratic Oath is not terribly complicated. It basically says, I will not furtively and maliciously hurt people in an abuse of my authority, and I will try to heal them when they are sick. I don't think it's a lot to ask that software engineers to agree not to create knowingly malicious software. It actually addresses exactly the problem you describe. If everyone has taken this oath, and adheres to it, there is no "someone else" to do that evil work.
Last point, there is a wide variety of software engineering work out there. Some of it may be mindless of the bigger picture of what is actually happening, but for any sufficiently advanced behavior to emerge out of a complex software product, some engineer at some level has to have some idea of the path they are going down to create or allow that behavior. And every engineer has the ultimate autonomy over how and what is created because it is our hands on the code. If you don't understand that, you don't understand the power of the profession.
No, I wouldn't say that. In many cases, management and engineering share the blame jointly and severally since they both have an opportunity to stop it.
> Said another way, engineers now need to be technical and legal experts in the business domain?
Engineers should know enough about their business domains to understand the ethical impacts of their work. Ethics and law are orthogonal, so thankfully this is generally much easier than being a legal expert.
> (Remember employees in the US depend on the company for health insurance. Saying 'no' could cost a lot more than just ones position.)
Thankfully, the healthcare safety net in much of the US is far better than it gets credit for, and the pay and availability of opportunities for software engineers in the US has generally been quite good. I'm sympathetic to this argument in general, which is one reason I don't think there should be an oath for, like, Amazon warehouse workers, but I'm far less sympathetic for anyone making 5+ times the median US income.
> Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself.
Your responsibility as framed to you by the business is the how, but upstream of the how is the question of whether or not to do it at all. If you contribute to a piece of software, you've tacitly answered yes to that question.
This would be more reasonable in the era of 10-20 years in the same company or industry. Needing to job hop every 2-3 years for a decent raise, and software skills applying to a vast array of industries, makes it less reasonable IMO.
> ...but I'm far less sympathetic for anyone making 5+ times the median US income.
Not everyone here or in software makes that kind of money. Some of us in the Midwest--or who aren't as skilled at negotiating--don't pull down nearly that much.
Yeah, to be clear, I don't think software engineers have an infinite level of responsibility for understanding the ethical implications of their work. If you were a software engineer at a credit rating agency in 2006, and you didn't see the ethical dilemma because you didn't anticipate that contagion would be exacerbated by the shadow banking system to bring down the global economy, you get a pass. But if your prospective employer is, like, locking children in cages, or spreading disinformation on political candidates, you should probably find that out during the interview process.
> Not everyone here or in software makes that kind of money. Some of us in the Midwest--or who aren't as skilled at negotiating--don't pull down nearly that much.
Good point - I'm also in the Midwest and make less than that, for what it's worth. I've naturally had FAANG in mind as I type these comments, and more generally I think salaries for the more unethical roles tend to skew higher.
Yep. That's more the responsibility of product managers, upper managements and chief architects/engineers.
I suspect a lot of philosophers would disagree with you that ethics is much simpler than interpreting laws.
Ethics forms basis on which law is built.
And, given that, it is not simpler to make ethical decision, it is harder. The decision should be worth being basis for a law, how's that simpler than following law?
Even if you take definition of ethical decision from Wikipedia: "An ethical decision is one that engenders trust, and thus indicates responsibility, fairness and caring to an individual." These words can bear negative connotations - if I beat people, I should be trusted that I will beat people, I should beat people fairly and I should care to beat an individual thoroughly.
Yes, I fully see you talk about principles. It can be seen that it is easier to make decisions from principles. But, you can misguide yourself about application of these principles.
Edit to add: If you are being asked to perform illegal or unethical acts as part of your employment, then perhaps termination is an ideal course of action? Unless of course your personal enrichment outweighs legalities or ethics in your worldview?
All these "companies take on a life of their own" arguments sound a lot like executives priming the pump of potential jurors with excuses. If decision makers cannot bear responsibility because of a company size or organizational structure then we can make some sizes and structures illegal before they stumble/march into devastating incompetence.
Was with you until this part. Just hold them personally liable if someone gets hurt should they create an uncontrollable system and predictably fail to control it.
Keep in mind the US already has laws around corporate structures and conflicts of interest. (Even if they're selectively applied.)
We heard the "too complex to understand" excuse a lot regarding the pricing of subprime debt. Except a lot of people did understand it was a problem. It's basically the "I'm too stupid to know what I was doing" defense. If we accept that defense and try to make regulation to protect them from failing (as was done in finance back then), we basically allow stupid people to continue to be in charge rather than being replaced as they need to.
Structures can and should be changed in this case. But shouldn't be outlawed.
The only argument that actually matters here is whether or not restrictions on corporate structure actually do violate freedom of association or not.
I'm reasonably skeptical that they do, given that the 1st Amendment hasn't stopped us from enforcing antitrust and monopoly legislation in the past. Yeah yeah, Citizens United and all that, but we regulate companies all the time.
But I'd still want an actual lawyer to weigh in on that, I wouldn't feel confident saying that there aren't limits on how far we can go in that direction.
> I'm reasonably skeptical that they do, given that the 1st Amendment hasn't stopped us from enforcing antitrust and monopoly legislation in the past. Yeah yeah, Citizens United and all that, but we regulate companies all the time.
> But I'd still want an actual lawyer to weigh in on that, I wouldn't feel confident saying that there aren't limits on how far we can go in that direction.
It doesn't necessarily hold that because one thing is legal, everything is legal. For example, we have 1st Amendment restrictions on threats and libel, but in the US hate speech is still protected speech. 1st Amendment exceptions are generally pretty narrow and specific in the US.
In the same way, clearly some corporate regulation is OK. It does not follow that there's literally no limit on what the government can dictate about how a company can operate. I would prefer to get input from a lawyer before asserting that so confidently.
I should phrase it differently. Why is an absolute freedom of association more important then the freedom from being harmed by large associations with amoral machinations. The original argument asks that if large corporations inherently obscure moral outcomes, maybe they are immoral, which is an argument that puts these two moral axioms in conflict. Simply stating that one side wins is thought terminating; its important to argue for why its better.
As someone who works in safety critical code nothing irks me more than when people absolve themselves by saying “I’m just a programmer, that’s not my job/problem”. We need to hold ourselves to a higher professional standard
If you're an actual engineer of any kind, you always have some choice on this. You make architectural decisions every day, and you generally work for places that do, in fact, take your input into consideration. If you don't work for any of those kinds of places, then you are still responsible because you wrote the code to enable it. You can always say "no". There are consequences for that, for sure. You can always quit as well. And it may still get made. But it won't be by you.
And sometimes, that's still better than the alternative.
On the contrary, a doctor's desision affects the life of an individual patient in a very clear and understandable manner.
Illegal stuff is illegal because not knowing the law is no excuse. For your own safety and good, you better have some grasp on legal stuff in your domain. Don't have to be an expert.
(Remember employees in the US depend on the company for health insurance. Saying 'no' could cost a lot more than just ones position.)
Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself.