Re:Ask HN: Best “I brought down production” story?
by andrewstuart2 on Sunday June 27, 2021 @01:03AM
(#27648756)
Having worked in a very large, bureaucratic company I can say that I strongly suspect that just ignoring systemic failures as learning opportunities is also not a sustainable. Too many times I’ve had to yield on something and say “I guess they’ll learn when this fails” only to see them easily move on or get promoted before the failure occurs. They don’t learn their lessons.
I suspect the solution is to find a way to make sure the consequences of the decision are fed back properly into the system directed at the right person. How to do that, I have no idea.
Re:Ask HN: Best “I brought down production” story?
by jfrunyon on Sunday June 27, 2021 @02:01AM
(#27648997)
It's not that you ignore them. But your first step if someone makes a mistake should not be to fire them. Maybe they need some coaching, maybe they have too much access or authority, but everyone makes mistakes. The key is whether or not they learn from them, or keep making them.
Re:Ask HN: Best “I brought down production” story?
by ketzo on Sunday June 27, 2021 @01:19AM
(#27648817)
A piece of the system (a junior developer) is allowed to make mistakes. The person responsible for architecting and protecting the system (the CTO)... less so.
Re:Ask HN: Best “I brought down production” story?
by d1str0 on Sunday June 27, 2021 @02:50AM
(#27649226)
That depends, this might have been this CTO’s first time as CTO. Without knowing the story, they could so have been pretty far out of their element and just lucky to be a founder or something.
Even C-level people always have to have their first day as C-level and of course they will make mistakes.
The important thing is learning from them of course.
You don't see the contradiction of terms there? A systemic failure is by definition not the responsibility of one person. You're saying people should be able to make mistakes. But not those people.