owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.
Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.
Something we tell employees during our onboarding in a technical comes down to this.
- reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.
If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).
Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.
2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.
In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.
My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.
To give you a counterexample.
The same company my prior example came from, also had some other "silly mistakes" made by an intern. He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.
Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.
Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still. Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).
The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.
In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.
Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.
Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.
The only way to guaranteed never have mistakes in your code, is to not have any code at all.
Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.
If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.
That outage is what drove us to rearchitect all of Netflix to be multi-region.
Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.
SolarWinds is looking for an Equifax not a Netflix.
Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong.
The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.
If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.
At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.
Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.
One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access.
One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.
[0] https://netflixtechblog.com/a-closer-look-at-the-christmas-e...