> something that is hard but not impossible to go wrong may carry huge amounts of risk.

I think it's the definition of the black swan theory[1].

[1]: https://en.wikipedia.org/wiki/Black_swan_theory

Fantastic take. Thank you.

Risk = likelihood * severity

That's also an approach, but it may lead to endless discussions about how likely something is. It's easier to tell what the worst possible consequence is (what this article calls criticality). After that it's fairly straightforward to figure out if you're doing enough to prevent this scenario from being realized (which is more like coverage = risk * mitigations).

Sometimes one has to include detectability as well.

Severity should include detectability. If you never detect an issue, it's not an issue because nobody sees it.

Usually it is a seperate factor, at least as far as P/D-MEAs are concerned. Quick and dirty, sure, it can be included in severity. Personally, I prefer the increased transparency and granularity of having detectability as a different factor.

To me, that’s a subcomponent of severity

Or, one cpuld alao just follow some standard processes all the time, instead of developing an individual approach every single time. Those standards should contain mitigation for most of the common risks, and rules to apply for the rest.

And one of those standards, an no I don't give a shit about developer experience, software or otherwise, should be you never ever test on production. As soon as you work on real products for real customers you better start behaving like a professional. Childs play is over as soon as some one is paying you to do stuff.

Every enterprise I’ve ever worked for, including corporate giants running the backbones of modern finance ran tests on production. Post deploy smoke tests, fail over tests, small group alphas, test migration/roll backs etc.

Not being confident in your test plan is a sign of immaturity not maturity because at some point you are going to need to validate how something behaves in production.

There are a wide range of processes, procedures and software architectures to get you to being confident that your production testing is doing more good than harm for your customers but in an environment where you can deploy new software you are going to do some testing in production.

There’s many reasons you test in production! The most recent TiP project I worked on was running a regression suite against production to notify outages before before users did (in an irregularly used but critical system, such that you can’t just go by user logs alone).

When I was doing disaster recovery we measured three things: 1) the likelihood of a particular scenario, 2) the magnitude of the impact if the scenario unfolded, and 3) the level of effort expected to recover. It was the combination of these three things that prioritized decision making.

We were working towards a business continuity plan which can include incidents like your main office and operations being destroyed and having to quickly relocate all services to 3rd-parties using off-site backups with minimal staff. While that was a worst-case, a primary focus was just getting a notification site up and running in the event of a network outage because that was vastly more frequent and had high visibility.

It was a very interesting project and I learned quite a bit about how to think comprehensively about the solutions we provided.

>> Something is only an actual risk if it hurts the bottom-line.

Sometimes not even that. We have see many huge breakdowns in recent years that did hit bottom lines, but didn't impact stock prices. Perhaps, at least for a publicly-traded company, the only real risks are those that might impact stock prices. That might include things that hurt other companies if doing so might result in money leaving the entire sector.

It's a really useful perspective in real-life scenarios when you're not developping critical software. Of course a baseline of risk-avoidance is always important, but businesses/custommers/users most of the time are ready to handle some risks, like downtime, bugs, delays, etc. SWE and developpers are the more risk-averse of the two parties, which leads to us over-valuing the importance of robustness and stability.

For example, it's way easier/faster to implement observability and some sort of rollback of bad versions than to try and prevent every possible way an app could crash and trigger a bunch of problems. What's going to happen if the app crash is pretty simple : customers will be mad (CS/Marketing/PR can handle them), you'll notice the downtime quickly and rollback (or maybe even rollback automatically!). Then you'll be in a perfect position to handle what went wrong : systems will be back on a known stable position and all the stress of trying to fix something in a live production system will be gone.

Of course, there is no magic bullet. Some problems aren't solved by rolling back services. (e.g. A thundering herd of clients caused by re-deploying an old build overloading your database.)

Yes of course, my fake situation was assuming a pretty boring case of failure with an easy out (rollback). The underlying principle is that most of the time trying to preempt every situation is way more work than being conscious of them and giving yourself and your team(s) reasonable tools to mitigate them :)

> may in fact be doing harm without actually sufficiently preventing any risk.

The biggest practical impediment to increasing velocity of delivery that I encounter is trying to convey this. People can visualize and estimate the risk and impact of a deployment gone wrong, but have a hard time estimating the impact of processes that slow down delivery. Therefore they overindex in heavy and "safe" processes (which often don't increase safety) at the cost of speed of iteration.

I'm not sure how to define this asymmetry, maybe some variation of loss aversion.

This was an interesting bit of math I did when I joined a startup. It's pretty counter-intuitive to think about how very large numbers can increase the importance of small ones.

Say the company you work for is worth $10,000,000, and that you're hosted on GCP. Now take your best guess: what do you think the likelihood is of e.g. a fire or earthquake or something occurring in all relevant Google infrastructure simultaneously*, basically ushering in the end of all of your infrastructure, data, and backups? Frame that in a number of years. Is this kind of event something that may happen once in a thousand years? Once in ten thousand years? Let's say this is the sort of thing that might happen once in ten thousand years -- that's a long time!

Then the cost of this particular risk to your company is $1000 / year.

This kind of math isn't just a toy. When you have questions like "would maintaining actual physical backups in a safe somewhere outside of GCP be worth it?", you now have a framework to answer them ("if it would cost less than $1000 per year, then yes")

--

* or substitute in your favorite company-ending event.

Similarly:

"We are spending 50$ per month just for one test in our code. We could cut it down to 10$ if we wanted."

"How many hours would it take to reduce the spend? If it's more than a couple of hours for a senior engineer, then it's not worth it."

We kept spending money on this inefficient test and it was the right choice.

Reciprocally, when low wages are available in manufacturing, comapnies are less likely to use automated proccesses because labor is so cheap it can cost less to eg. have two buckets/wheelbarrows between two parts of a factory line with one person to swap them rather than use a conveyor belt. Getting 100% automation would allow factories to come back to the US but getting that last 20% at a competitive cost is difficult. See America’s largest tool company couldn’t make a wrench in America (wsj.com).[0]

[0] https://news.ycombinator.com/item?id=36828861

This is similar, but one of the benefits of thinking about 'things that musn't happen' and relating them to 'things that can go wrong and how to prevent them' is that it avoids talking about expected damage.

This avoids two nasty problems with trying to express risk as an expected value.

The first is that it is hard to express all kinds of probabilities and damages numerically, not all kinds of damages convert easily to money, and some probabilities are hard to guess (you quickly get uncertain probabilities, but expected values just flatten those into an average again). Even without those issues pinning a number on it can lead to lots of discussion (good if you want discussion, not so good if you want to get shit done).

The second is that you easily fall into the trap of assuming everything has an average, and that the law of large number applies. While physics kind of helps you there by putting hard limits on the maximum amount of damage possible, you may end up in a situation where all nasty stuff is in the long improbable tail. Good example is earthquakes, magnitude increases tenfold for every point in the Richter scale but frequency also only decreases tenfold, what then is the average?

Well and something that's not really a big problem, but worth thinking about, some of these eventualities may very well cause you damage but are beyond your sphere of influence. Sure you should try to avoid going bankrupt if someone knocks over a server rack, but if all google data centres go down over an entire continent you've got bigger fish to fry. So focusing on the things you can do something about is a helpful way to keep focused.

While I agree with your math, I am curious how much your physical backups would be worth if something so catastrophic occurred that all of Google/AWS/Azure cloud services were destroyed. Whether that be an act of war, a massive solar flare, etc., I am curious if it would even matter anymore that you had those backups.

Very well said.

I had a similar conversation as a new-ish fractional CTO last year. One team was working on a new CRM product that was effectively alpha-level software used only internally. The team had become terrified of shipping and breaking something and was horrifically risk averse. For a new release that the team was going to delay again at the last minute, I got the CEO on the release call and asked him what would happen if the release completely failed and it took us an entire day to get the product working again. He replied “Not a big deal. The users would just write stuff down like they do today and key it on tomorrow. It’s not like this has enough features to be critical or anything”.

The team was completely stunned. It goes without saying we did the release, found a small mistake, fixed it, and life went on.

Teams really do have understand who her users are and criticality of the software.

This story perfectly aligns with the arguments in this article. I'll add it as a note if you don't mind.

Please do :-)