99% of folks here don’t deal with life and death matters. When you do, you get a lot of practice at canceling, which helps. There’s also the fact that you can say, “if you do this, people will die,” which is a pretty bright line when you’ve got something to back it up. I actually found it more difficult to take a stand in less consequential fields. But it’s still very, very hard to look someone more powerful in the eye and tell them no. I’ve had to do it and it’s not fun. (Nothing on the scale of a Shuttle launch. I once had to effectively revoke a judge’s flying privileges, for instance.)

McDonald is on a whole other level though. That’s not just your employer, that’s the entire space program. I hope I never have to find out whether I’m made of the same stuff he was.

>if you do this, people will die

In my experience, this is not how it works. Very rarely can you say definitively that “people will die”. Reliability is shrouded in uncertainty and the best you can say is, “people may die, eventually ”. If you make too absolute of a claim and it didn’t occur you will gradually erode credibility. Look at Columbia... they knew the foam shedding was out of spec but had so many instances of probability being on their side, if you claimed people would die act each of those launches soon you just become noise. Over a long enough timeline you may be right, but phrasing in that way is unlikely to help in low probability events.

aka Normalization of deviance https://en.wikipedia.org/wiki/Normalization_of_deviance

And a short video in context of challenger accident : https://www.youtube.com/watch?v=tC0qkVf5SmE

Mullane was on STS-27 -- that was a disaster on the level of Challenger or Columbia but by pure luck they avoided a deadly ending. The failures were predictable. And this was the second flight after Challenger's last.

I think the underlying point is that there are cognitive biases that prevent decision makers from effectively understanding that risk. It’s very much like financial crashes; after the fact it becomes obvious the risk was there but our biases prevent most people from seeing/understanding it effectively in the moment

This man understands. It’s the .001% chance that kills people

There was an anecdote from one of Czech researches in Papua-New Guinea who said he was surprised that local hunters absolutely refused to sleep under suspicious, not entirely healthy trees, and were very careful about examining the spot before they chose it for rest. He considered the chance of the trees falling down very low, perhaps less than one in thousand.

He changed his attitude when the elders explained to him that while the chance was very low, the hunters took a lot of naps in the forest throughout their lives and over decades, the low chance translated to almost-certainty.

I think I read this in a column of Jared Diamond's.

No, it's the 10% that kills people. They knew the O-rings were failing because they had analyzed previously discarded rockets. So the engineers said, "We got lucky this time; eventually this is going to kill people." But after launching 4-5 times and having O-ring failures with no problems, management (and probably everyone) started thinking it wasn't really as dangerous as it was.

The percentage loses meaning without an expected number of opportunities to calculate the expected value. 10% risk x 1 flight might be deemed acceptable. 10% risk x 100 flights means you should plan on losing 10 shuttles. Neither statement about risk probability is wrong (0.001% vs 10%), but both are incomplete. The actual shuttle risk was calculated between 1/78 to 1/254, if I remember correctly

It's enough that people may die (or even get injured) as a consequence of your work. That's the line you shouldn't cross and it actually helps to have such a clear and unambiguous definition of that line.

I understand that stance, but when humans are the decision makers and have all kinds of other pressures (cost, schedule, etc.) it’s often not enough that people “may” die. Especially if we’re talking about risky endeavors like human-rated space travel, “may” is only one level across the gradient, and that level could be considered an acceptable risk by some. You will never drive risk down to zero, so “people may die” will almost always be true. If that’s your threshold, you will never launch.

The real difficulty is accurately gauging risk, especially when there isn’t substantial data.

Sts-27

https://youtu.be/BswkvaAaqSM

1988. Damage to orbiter Atlantis.

Not the same foam that shedded (pieces from SRB rather than external tank). But still demonstrated failure modes.

The failure mode was known (hence why it was considered out-of-spec) but it didn’t translate to an accurate risk assessment. It was deemed “in family” meaning it was known to be outside of spec but not a risk. Somehow, over many flights, something that was originally characterized as a safety risk lost its credibility as a risk. It’s not that it was unknown, but the understanding of the risk was changed. I wonder if STS-27 led to a false sense of confidence

I'm curious to hear what you think about the Therac-25 situation because the line between a code change and fatalities isn't as clear cut as with Challenger, but definitely still there.

I took a hard line on an ethical issue at work. It crushed me. I left when it got done anyway. I'm never working for a startup again.

Hey, whatever it was, I appreciate people like you are out there regardless of whether the thing got done anyway.

I'm in currently in the valley of death phase of startup life working on a tiny piece of the climate problem, but when the day comes that I'm in a position to hire teams I want them to be able to look me in the eye and oppose me when I'm going off the ethics track. How we got here to needing the climate problem fixed was people making their peace with consequences of their actions, and it's not going to be solved by doubling down on that approach.

In my experience big tech is no more ethical than startups. I respect where you are coming from though, there are plenty of other reasons startups are hard. Much of the ethical tapestry at a startup is on the founders.

I’ve done it too. So it’s you + me + McDonald + ...

You know, for all its evils, it is sometimes surprising that the world is relatively peaceful (for 60% of the globe), full of donators, people fighting for women, people refusing to participate to bad schemes. Even in opposing views, most people are doing it because they believe the world will be better like this. Also, absolute stupid people have always been the norm, and the bad ones have always been assertive and powerful and yet, the world is not so bad, so we are on a good streak.

me too. IP issues - founders stealing engineers ideas, scanning their email for IP but not naming them as inventors in the patent filings.

Karma is a bitch, has a long memory, and delivers retribution.

Move and fast and break things makes perfect sense when you're making a dog walking app and need to win time to market.

The mindset is different when building truly critical systems.

There's not really anything unethical about moving fast and accepting the risk that comes with that for most of the things people are building here.

Well, dog walking apps have pretty major safety issues. They've had problems with crime, serious injuries, pets have died, and all sorts of lawsuits documenting it.

Your broader point is correct, I'm just not so sure everyone is as good at judging where that line is as they think they are.

Yet many here cheer for Tesla, Über etc in the self driving space for exactly this attitude, because it they think it is necessaty for disruption.

> It's one thing to praise a hero like him... but how can we be that guy while having a stable job?

You decide that ethics and life are more important that your next few paychecks years before the event that makes you stand up and speak out. Sort of the stoic version of dress for the job you want.

For many businesses, from the regular employee perspective, a lot of the hard ethical questions unfortunately got answers and built into policy years agoz often well before platforms and products got big for those that succeeded. Getting old discussions resurrected is and likely will remain a hard problem in human organizations forever.

Perhaps counterintuitively, working on systems with clear life safety concerns often makes it easier since there are very clear consequences. It's one of the reasons for very harsh regulatory penalties, to be big and shocking enough for those that don't have strong ethical paradigms. In the industries I deal with there are several "million dollar per day" fine structures I hear people going on about that I never correct with more accurate information because if they're oblivious enough to not understand they're also oblivious enough to need that fear.

You can't, you have to be galactic citizen and do the right thing or best we end up like the Borg.

If you want to remain true, you have to have FU money and know where that line is. Some peopled don't need that moral cushion, others do, it isn't a value judgement. We need to construct a world where Allan McDonald's can flourish.

https://youtu.be/FQDe8Y9BBMo?t=100 Be the foundation for the future.

I think people with this kind of integrity are very few in general. Being HN participant does not really increase the chances. Might even be the other way around as being smart asses it is easier for us to come up with the reason to keep our conscience quiet.

I do not think it has anything to do with "move fast and break things" way of doing things though. You can't expect the same approach and investment in safety in generic company vs some nuclear power plant or likes.

I've never been faced with a decision so grave as McDonald, but I've done my best over the years and I'm content. There are lots of opportunities to move the needle ethically that don't require sacrifice at all. At least one time the organization actually changed — credit due to the people who listened with open ears.

(Throwaway account because making a difference doesn't necessarily mean making a public show.)

Props to you.

>I can't help but wondering how many people today on HN could make such a decision, especially those who insist to "move fast and break things"

This gets into the argument about whether people who write software should be called engineers. There is good reason why nearly every engineering body out there has something along the lines of "Hold paramount their duty to public welfare" as the first item in the code of conduct. "Move fast and break things" is pretty much incompatible with engineering.

I think it’s kinda like how once digital photography became good enough you didn’t have to worry about wasting film. You can take 20 pics and later pick the best one, rather than spending 20 minutes trying to get the perfect shot.

With software how it is today the “outer software” engineering has created a little virtual realm where things can go haywire and fail but it’s in a mostly padded room. And of course it’s all running within very well regulated and stable hardware.

Maybe coding itself isn’t engineering anymore than welding or running cables is but both computer and software engineering was required for coding and to make the code do anything significant it takes some engineering, or you could just start welding shit together!

I disagree

>There is good reason why nearly every engineering body out there has something along the lines of "Hold paramount their duty to public welfare" as the first item in the code of conduct. "Move fast and break things" is pretty much incompatible with engineering.

Those aren't mutually excluisive.

Move fast and break things when it comes to crud apps is totally different thing.

You can also use "move fast and break things" in order to achieve "Hold paramount their duty to public welfare"

Move fast and... does not imply unsafe.

Move fast and break things _absolutely_ implies unsafe. It doesn't guarantee unsafe, but it most definitely implies it.

The word "break" is in the phrase...

Yea, I don't think I've really been under that gun either.

About the closest I came was when working in telecom, for a new tech deployment I had my hands pretty deep in the lab environment, so when things were broken in the lab, bad config, etc I would get pulled in troubleshoot and solve the problems.

Well one day 911 wasn't working in the lab and the problem got thrown my way, and it wasn't an obvious problem like someone miss configured something or broke some config somewhere. In telco at the time it was all vendor driven solutions, so I intentionally left the system broken to bring in the vendor to troubleshoot, and it was clear, this is a lab, let's not treat it like production and as such we don't need immediate recovery, we want to get to the root cause so it doesn't happen in production.

The next day, the handset verification team was on me, saying they need this to work immediately since they need to validate some device by such and such date. And I basically said listen, there's a software problem in this product, and we don't want it to go to production. And if I don't get it fixed it could blow up in production on us. I also told them if I don't make progress in a day or two, I would try and reconfigure another environment for them so they would get unblocked, but otherwise was not willing to just reset this system so the problem went away.

I was also doing my own investigation as much as I could since the vendor wasn't always the most reliable, and I encountered something unexpected. It looked like a node was rebooted, so I tracked that down, and found a senior architect who new I was working on solving the issue had rebooted one of the blades. His answer was basically the device team was complaining so he just went in and rebooted the node so they would stop complaining to him.

Luckily, he didn't know enough on how to really reboot the system, so it just synced back with it's backup and still had the problem for us to investigate.

The vendor comes back and goes ah yea, the 911 handler is using the wrong memory region for storing emergency calls, so instead of being able to allocate a hundred thousand records or whatever it was for active emergency calls it was using an administrative region that could only allocate something like 5 calls. This was enough years ago that I forget the exact number, but it was less than 10. Not just that, but there was a second bug, a certain 911 call flow would allocate the call but not release it, which is why we couldn't make any 911 calls in the lab, we had leaked all of the reserved memory for emergency calls.

I just remember being so livid, because the culture for anyone who dealt with that system was it's failure is just in their way, so lets just escalate and try and make it go away so we can continue on with our jobs.

And it would've been so easy to just reset the whole thing so that people would stop complaining. It was just a lab after all.

That's terrifying.

When you work in the industry writing software which can kill people, it is not so difficult to raise up your voice and say no when it could threaten life of people. Even if you are wrong it is still useful to raise your concern, as this would trigger an in depth analysis. I would say that it is even easier when your company experienced deaths due to products they make.

But.. I am not in the position of being responsible for signing the design which makes my life easier. If an accident happens I have my conscience for myself and proof that I objected.

However, it is much more difficult to gain attention when you cry foul to something unethical being done (diesel gate comes to mind here), even though it can remotely leads to deaths.