We used to have an issue of deployments breaking in production, and one of the reasons was that we did not have some kind of smoke test in the post deployment (in our case we only had a rolling update as a strategy).
The rational solution was only create that post-deployment step. The solution that our German managers demanded: cut access to deployment for the entire team, “on-call” to check the deployments, and a deployment spreadsheet to track it.
It leads to situations where you need a certificate from your landlord that you take to the certified locksmith that your landlord contracted and show a piece of ID to order a key double that arrives 3 business days later at a cost of 60€. A smart German knows that there’s a locksmith in the basement of a nearby shopping mall that will gladly duplicate any key without a fuss, but even then the price is inflated by the authorised locksmiths.
I document German bureaucracy for immigrants. Everything is like this. Every time I think “it can’t really be this ridiculous, I’m being uncharitable”, a colleague has a story that confirms that the truth is even more absurd.
It’s funny until you realise the cost it has for society at large. All the wasted labour, all the bottlenecks, and little to show for it.
When Grandpa was 20 years old he left the house and forgot to take his keys, so every time he left the house he checked his pockets for his keys.
When he was 24 he left the house and left the stove on. He learned to check the stove before leaving the house.
When he was 28 he left his wallet at home. He learned to check for his wallet.
...
Now Grandpa is 80. His leaving home routine includes: checks for his keys, his phone, his wallet. He ensures the lights are off, the stove is off, the microwave door is closed, the iron is off, the windows are closed in case it rains...
Grandpa has learned from his mistakes so well that it now takes him roughly an hour to leave the house. Also, he finds he doesn't tend to look forward to going out as much as he once did...
> We then need to lean on data and experience of what the trade offs of those changes would be.
As engineering leaders, this is a key part of our job. We don't just blindly add processes to prevent every issue. I should add that we also need to analyse our existing processes to see what should change or is not needed any more.
However, you say you're agreeing with scott_w, and scott_w is criticizing the article. So this is confusing.
I'm going to strongly disagree with this (when it's done well, not bureaucratically).
We review what can be improved due to problems and we incorporate it into our basic understanding of everything we do, it's the gaining of experience and muscle memory to execute fast while also accounting for things proactively.
It's a long term process but the payoff is great. Reduced time+effort on problems after the fact ends up long term increasing amount of valuable work produced.
The key is to balance this process pragmatically.
The answer this post gives to that bizarre question that always gets asked, is ‘nothing’, unless you want to significantly adjust the speed that we deliver features.
Any added process or check is going to impose overhead and make the team a little bit less happy. Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between.
In extremis, you’re reduced to a situation in which you have zero incidents, but you also have zero work getting done.
On the other hand, enforcing a manual external QA check on every release WILL slow things down.
You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.
I agree with the premise here, but in my experience running incidents review the issue that I see it’s a mixture of a performatic safetycism with reactivity.
Processes are the cheap bandaid to fix design, architectural and cultural issues.
Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone.
They can be, yes. I have a friend that thinks I'm totally insane by wanting to release code to production multiple times a day. His sweet spot is once every 2 weeks because he wants QA to check over every change. Most of his employers can manage once a month at best, and once a quarter is more typical.
> Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone.
I 100% agree with this. Your comment also reminded me to say that incident reviews are necessary but not sufficient. You also need engineering leadership reviewing at a higher-level to make bigger organisational or technical changes to further improve things.
Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.
As far as I can tell, the author doesn't really give any generalized advice on how careful you should be, he's just pointing at the "carefulness dial" and telling people to make an informed decision.
there's a tradeoff on shipping garbage fast which won't explode on your hands and getting promoted. and there's also the political art of selling what you want to other people by masking it as what they want.
you and i and most people here will never understand any of that. good luck. people who do understand will have the careless knob stuck at 11.
these analogies help us rational people point out the BS at least, without having to fight the BSer.
Your comment starts with an ableist slur so I’m sure it’s going to be good /s
> you and i and most people here will never understand any of that. good luck. people who do understand will have the careless knob stuck at 11.
Nah, reading this comment wasn’t worthwhile after all.
> these analogies help us rational people point out the BS at least, without having to fight the BSer.
How cute, you think you’re “rational.”
What is your objection? Oversimplification? Lack of utility?
Same energy here. “Be more careful” is extraordinarily hand-wavy for a profession that calls itself engineering.
That’s not how our jobs work. We don’t “adjust a carefulness meter.” We make conscious choices day to day based on the work we’re doing and past experience. As an EM, I’d be very disappointed if the incident post mortem was reduced to “your team needs to be more careful.”
What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. We then need to lean on data and experience of what the trade offs of those changes would be. Asking a test? Go for it. Adding extra layers of approval before shipping? I’ll need to see some very strong reasons for that.