News for hackers. Stuff that matters.
Re:The Problem with ‘5 Whys’ (2016) by CobrastanJorji on Wednesday April 10, 2019 @12:44PM (#19626446)
Many years ago, I worked at Amazon, and at the time they LOVED "5 Why" analysis. I had to write a couple of them. The second and final one I wrote came back from my manager, who told me that it was unacceptable for my "5 Whys" to blame something outside our team's control when the most immediate cause was a bug written by our team, and I needed to come up with a different ultimate cause. This was really frustrating, but it was easy enough to completely change the cause without being dishonest. "5 Why" style analysis is really easy to steer toward whatever you want to blame.
Re:The Problem with ‘5 Whys’ (2016) by awill on Wednesday April 10, 2019 @01:07PM (#19626687)
You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. If you depend on something, you need checks, alarms, and whenever possible the ability to handle dependency failure well.

Imagine you're built on a public cloud in a single Data Centre. That DC has an issue. Your 5 Whys shouldn't end with 'the Data Centre had an issue' It should continue with 'my service was only in one DC.... Why?' and continue from there.

Re:The Problem with ‘5 Whys’ (2016) by perlgeek on Wednesday April 10, 2019 @02:46PM (#19627609)
> You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service.

Our service was down.

Why?

Because the platform it runs on went down.

Why only one platform?

Because management doesn't want us to spend several years engineering for multiple platforms.

---

I find it hard to imagine a realistic path of 5 whys that leads us back to a cause within our own team.

Re:The Problem with ‘5 Whys’ (2016) by Karrot_Kream on Wednesday April 10, 2019 @03:31PM (#19627989)
> Because management doesn't want us to spend several years engineering for multiple platforms.

Yes and that's fine. It's okay to end up with an RCA like that and note that as a known risk. If a cost-benefit analysis shows that the risk is dwarfed by the solution, then it's not worth fixing. Knowing the existing risks of your system and reevaluating these risks in the face of changes (more people available, inefficient context switching, etc) is a key component of making progress.

Re:The Problem with ‘5 Whys’ (2016) by awill on Wednesday April 10, 2019 @03:51PM (#19628169)
True. In a case this simple, you likely wouldn't have to write an RCA.

However, most real life situations I've seen show that there's always room to improve. For example, how quickly did your service recover once the underlying platform recovered. Rare events (like DC failure) can be a bad thing for operational training. Teams forget how to recover. No runbooks, no training, or tooling that's infrequently used no longer works.

Re:The Problem with ‘5 Whys’ (2016) by grogenaut on Wednesday April 10, 2019 @07:21PM (#19629648)
You actually want to assign to root cause and track that. Them over time you build up data on if it's worth it to stay on one platform or not. Eg doing this let's you build a risk profile to counter the $/time argument against. Then you get to make an informed decision.

You don't just shruggie and move in "that ship has sailed" at least if you're doing it right.

Re:The Problem with ‘5 Whys’ (2016) by msla on Wednesday April 10, 2019 @04:35PM (#19628525)
Alternatively:

Our service was down.

Why?

Because the platform it runs on went down.

Why only one platform?

Because management declined to approve purchase orders for sufficient hardware to allow redundancy.

---

And your career just hit a wall. Passing the buck up the chain tends to do that.

Re:The Problem with ‘5 Whys’ (2016) by sfifs on Wednesday April 10, 2019 @05:06PM (#19628781)
So you don't word it in a way to assign blame. You say previous cost benefit analysis indicated X known trade off from lack of redundancy. This incident either confirms the trade off (finance will usually support you for this) or say additional X factor not previously known needs to be taken into the trade off and revised.
Re:The Problem with ‘5 Whys’ (2016) by LaGrange on Wednesday April 10, 2019 @10:05PM (#19630526)
Plot twist: X factor was known but dismissed by management. Management knows this and knows how to read between lines. Your career hit a brick wall and you didn't even get to openly snub the management.
Re:The Problem with ‘5 Whys’ (2016) by roenxi on Wednesday April 10, 2019 @10:56PM (#19630780)
You're pushing the scenario into foolish territory. If management decisions are the root cause of a problem and they won't accept that, then the quality of the process used to identify the root cause is irrelevant. You could use the best or worst process and it wouldn't matter.

If your management is bad, then there are a lot more risks to your career than trying to work with them to improve the process. It is more likely that a given employee has dead-ended their career because they don't know how to explain a problem without blaming people and having the humility to see from other people's point of view; both useful skills for getting a promotion.

Re:The Problem with ‘5 Whys’ (2016) by mesozoic on Thursday April 11, 2019 @02:25AM (#19631661)
Interesting. I'm certain you're right don't blame your superiors if they hold your fate in their hand. But what do you do when you know they are incompetent? Cover their ass and feed their incompetence for your own gain? I'm not sure.
Re:The Problem with ‘5 Whys’ (2016) by dagw on Thursday April 11, 2019 @05:27AM (#19632379)
But what do you do when you know they are incompetent?

Is your superior's superior also incompetent? What about his superior? Does your superior have someone else on his level that isn't incompetent? If you genuinely believe the company is top to bottom incompetent then what are you still doing there? Otherwise identify the competent people and start working that angle, either to get yourself transferred under someone more competent or to move critical decision making power from your superior to someone more competent (perhaps yourself)?

Or, if you're that sort of person, find a way to use their incompetence to mask your lack of giving a fuck and just slack and collect a pay check until they fire you.

Re:The Problem with ‘5 Whys’ (2016) by therein on Wednesday April 10, 2019 @07:35PM (#19629733)
That indeed sounds much better.
Re:The Problem with ‘5 Whys’ (2016) by 8note on Friday April 12, 2019 @06:27PM (#19649609)
how do make effective action items with that root cause?
Re:The Problem with ‘5 Whys’ (2016) by danso on Wednesday April 10, 2019 @01:40PM (#19627011)
This reminds me of the Twitter [0]/HN [1] thread by someone trying to argue that the Boeing 787 MAX couldn't be blamed on software engineers, because it was the aerodynamic/mechanical/systems engineers and business executives who demanded too much (and also, the pilots for just not being better trained). Yes, that's all technically true. We could even put more blame on the human desire for convenient speedy travel as more of a direct factor. But software engineering is part of systems engineering for any modern project. We should want and expect software engineers to have the agency and bigger-picture thinking to do more than just design for inadequate inputs.

[0] https://twitter.com/trevorsumner/status/1106934369158078470

[1] https://news.ycombinator.com/item?id=19414775

Re:The Problem with ‘5 Whys’ (2016) by jdietrich on Wednesday April 10, 2019 @02:44PM (#19627595)
The buck stops with senior management; the buck always stops with senior management. That is the essential lesson of the 5 whys and the essential problem with trying to use it without being fully committed to the Toyota Production System. If senior management aren't willing to cop the blame for literally everything that ever goes wrong, the whole effort is just a dog-and-pony show.

https://en.wikipedia.org/wiki/Toyota_Production_System

Re:The Problem with ‘5 Whys’ (2016) by dtech on Wednesday April 10, 2019 @02:48PM (#19627635)
I disagree. If an outage in the product your team managed was caused by a bug your team introduced 5 can point at several root causes and solutions that your team will be able to control and implement.

Management only comes into play when causes like "under-funding", "constantly changing requirements" or "too tight deadlines" are identified.

Re:The Problem with ‘5 Whys’ (2016) by jdietrich on Wednesday April 10, 2019 @03:28PM (#19627962)
Management have ultimate responsibility for your team - staffing, training, resources, workload and everything else. If one of those is wrong, it's their fault. The alternative is pure scapegoating. Blaming a team or an individual doesn't get to the root cause - why did that employee underperform, why was that team badly led, why did nobody identify or report these problems until it was too late? The answer might be as simple as "we hired a complete moron", but management have to take ownership of that error if they want to prevent similar errors in future.
Re:The Problem with ‘5 Whys’ (2016) by Darkphibre on Wednesday April 10, 2019 @05:25PM (#19628946)
Reminds me I need to finish Extreme Ownership. Got about half way through, a really excellent philosophy.
Re:The Problem with ‘5 Whys’ (2016) by vajrabum on Wednesday April 10, 2019 @03:19PM (#19627881)
Management also has to support the idea of blamelessness and the costs of investigation and remediation and to participate in reviews. Without that support it may work some of the time but often won't.
Re:The Problem with ‘5 Whys’ (2016) by bluGill on Wednesday April 10, 2019 @04:21PM (#19628412)
Management is still at fault for not putting in place enough quality to ensure that the bug didn't happen, or was found first. There are a number of ways to do this. They can spend more time and money testing the product before it goes to production. They can spend more money/time on automated testing infrastructure. They can spend time/money on failover to last good version. They can spend time/money on up front design. They can decide that the costs of any of the above is too high and so they will accept the risk. Depending on your exaction situation any of the above can be most cost effective (often a combination - in particular accepting risk eventually is required no matter how much you spend on something else).
Re:The Problem with ‘5 Whys’ (2016) by eridius on Thursday April 11, 2019 @01:08AM (#19631307)
The fundamental point of the argument that software wasn't to blame was that the problem wasn't with the way the software was implemented, but with the way it was specced, and that it was performing to spec, and therefore blaming the software was pinning the blame on the wrong folks as the software folks didn't write the spec (and the people who wrote the spec didn't control the factors that led to the need for the spec, etc).

> We should want and expect software engineers to have the agency and bigger-picture thinking to do more than just design for inadequate inputs.

Saying software engineers should have bigger-picture thinking doesn't help if the software engineers aren't empowered by management to make the necessary changes.

Re:The Problem with ‘5 Whys’ (2016) by CobrastanJorji on Wednesday April 10, 2019 @01:12PM (#19626734)
Sometimes a problem that affects you is outside your immediate ownership, and the right answer is to escalate and seek to fix it. If you limit yourself to only considering changes that your immediate team can make on its own without consulting others, you're doing your organization a disservice.
Re:The Problem with ‘5 Whys’ (2016) by awill on Wednesday April 10, 2019 @01:23PM (#19626843)
sure, if your service behaved as well as possible during your dependency's outage, yes.
Re:The Problem with ‘5 Whys’ (2016) by rakoo on Wednesday April 10, 2019 @01:26PM (#19626874)
Where does it stop though? Because in that case all externalities will become a "I trusted that dependency but it failed, why did I trust it and didn't work around it?".
Re:The Problem with ‘5 Whys’ (2016) by kevingadd on Wednesday April 10, 2019 @01:57PM (#19627181)
the causes of failures that you explicitly enumerate which were under your control aren't always You Did A Bad Thing.

For example, "I trusted this dependency which broke" doesn't mean you Did Something Bad, it just means that the cause of the failure was the lack of a way to validate the dependency and fall back on a known-good version. A good example of this is build pipelines relying on the ability to download npm/nuget/debian packages from a remote server instead of vendoring them locally - this is an intrinsically unreliable choice to make. The remote server being unreachable is not your fault, but there are steps you can take to fix this problem in the future. Avoiding external blame in your 5Ys (without ignoring the external factors) encourages you to make local changes.

As always you have to apply reason here and not adhere to dogma, but it's a useful dogma.

Re:The Problem with ‘5 Whys’ (2016) by dtech on Wednesday April 10, 2019 @02:52PM (#19627668)
It's okay to conclude after a root cause analysis that the root cause is not something you want to work to prevent, because the cost-benefit is skewed.

For some products a AWS S3 [1] outage might be a influential enough that they add a redundant second storage at another provider, for other it might be "this is too expensive and time consuming to work around for the problems it caused, let's hope Amazon gets their shit together"

[1] https://aws.amazon.com/message/41926/

Re:The Problem with ‘5 Whys’ (2016) by mikekchar on Wednesday April 10, 2019 @09:26PM (#19630324)
I think there is an underlying misunderstanding with the application of "5 whys" which is a bit like the misunderstanding that people have with prioritisation. Let me diverge a bit with prioritisation: a lot of people know about the story of how to fit large rocks, medium sized rocks, small rocks and sand into a jar. You start with the largest size and work your way down to the smallest size (sand). The smaller rocks fit in the spaces of the large ones. The moral of the story is to look at the items that are most important as being large rocks and the ones that are being least important as being sand. And by doing the most important things first, you'll get everything done. Because... um... lesser important tasks fit into the spaces of more important tasks??? What??? No. This is a bad analogy that looks like's related, but isn't related in any way!

Getting back to the question at hand. "5 whys = 1 how". Because if we do the root cause analysis, we will definitely find a solution to our problem. What???? Because there is always a solution to our problem??? It's that same skipping over of the most important detail: you aren't doing the most important problems first because that way there will be space for the lesser important problems. You are doing the most important problem first because then when you run out of time, you've done the most important thing! Similarly, you aren't finding a solution when you are doing root cause analysis, your are finding problems. 5 whys is a tool for helping search the problem space. That is all.

The "5 whys = 1 how" quote is really unfortunate and I think it's taken completely out of context. You don't just do 5 whys once and say, "Oh now we know how to fix our problem". You keep doing the 5 whys over and over and over again so you can identify your problems. And at that point you can start to get a handle on your solutions. "5 whys = 1 how" doesn't mean there is only "1 how" that you need.

And just to sum up, if one of your 5 whys ends up with "An external entity failed us", then you might need to find a way that you aren't relying on that entity. However, run some other 5 whys to help you search the problem space a bit more. You may find another "how" that will help you better.

Re:The Problem with ‘5 Whys’ (2016) by danbolt on Wednesday April 10, 2019 @01:40PM (#19627020)
I think what you're touching on is the crux of the article too. Depending on the "why" that gets delivered, you end up with very different conclusions.
Re:The Problem with ‘5 Whys’ (2016) by empath75 on Wednesday April 10, 2019 @01:44PM (#19627046)
But that would be 6 whys.
Re:The Problem with ‘5 Whys’ (2016) by BeetleB on Wednesday April 10, 2019 @03:25PM (#19627935)
>You can't end a 5 Whys with 'another team I depend on screwed up'. You have to own your software or service. If you depend on something, you need checks, alarms, and whenever possible the ability to handle dependency failure well.

On the big picture, I agree with GP - different people will come to very different conclusions with the "Why" game - because the world is complex, and many factors come into play when something goes wrong. You don't need to fix all of them, but it does become a game of figuring out the most convenient aspect to fix (and no, another level of "Why" won't fix that).

Somewhat related: I had a 2nd level manager who in meetings kept saying "In an ideal world, ..." and using that as a starting point for requirements, problem solving, etc. At one point, I interrupted him and said "Nope. In an ideal world, I would not need to work for a living. Your ideal is already far from it, so let's not make distinctions between ideal and non-ideal."[1]

It's easy to use the "whys" to come up with answers like "Because X needs to make a living" and "Because the incentives with the manager/department/org/company create a situation where testing is devalued (despite all claims by management to the contrary)" or "Because we cannot retain talent due to our compensation policies" or "Because people in the team are unwilling to learn version control systems newer than cvs."

These are not facetious answers - they should be treated on an equal footing with technical solutions. In my experience, management that does not want to deal with solutions that involve other teams/people/policies are ones where the work has been miserable. It's also been my experience that SW people seem to prefer technical solutions over alternatives, which is tragic. When you study things like negotiation, it's almost inverted. They have their own push for "Why", but the push is usually in the motives direction. The person you're dealing with is presenting a concrete demand that may seem objective, but behind it is almost always a fairly human reason (not looking bad, higher status, etc). And they always emphasize trying to root cause to those and addressing them.

Many here know the saying: In the top tech companies, all problems are social ones. They have the talent to achieve anything. If any piece of SW fails, it's not because they didn't have technically capable people. It's because they failed in the social/organizational domain. So when I hear management say they don't want to address that in the 5 Why's, I see management that is solving the wrong problem.

I took a systems engineering class once. The thing they emphasized throughout is "If every aspect of a product is designed perfectly as its own unit, you'll get a crappy product that will not win in the marketplace." The idea is that a perfect widget in isolation may not integrate well, and there are budgetary issues as well. There has to be a certain amount of compromise to integrate with other widgets in the product, and the teams involved need to talk to one another and have dependencies on one another. The role of the system engineer is to oversee that this is happening.

Insisting that your SW or service must be bulletproof of other services, when both are related and in the same product, is highly suboptimal (as is the other extreme). Often the optimal solution is to say "Our service will work, provided service Y works".

[1] I was on good terms with him so I got away with it.

Re:The Problem with ‘5 Whys’ (2016) by jimbokun on Wednesday April 10, 2019 @01:21PM (#19626820)
I don't understand, if the bug was written by your team, how could the ultimate cause be something outside of your team?

Seems like 5 Whys in that scenario would lead you to look at things like insufficient test coverage, no code review, code reviewers missed the bug because...

Those paths of inquiry lead to your team changing things under your direct control that could make it less likely for similar bugs to reach production in the future.

Re:The Problem with ‘5 Whys’ (2016) by espadrine on Wednesday April 10, 2019 @01:59PM (#19627196)
Broken expectations are a cause.

For instance, when ext4 gained traction, some users found a number of zeroed-out files after a crash.

Why? Because the programs managing those files updated them by truncating them and then writing them.

Why? Because fopen() has no "replace file" mode, even though the "w" mode seems to do just that as long as the system doesn't crash (it does O_WRONLY|O_CREAT|O_TRUNC).

Why? Because file systems used to not have delayed allocation when fopen() was created. ext4 introduced the concept to the ext family.

Why? Because ext4 was designed to improve performance compared to ext3, and that change does that without breaking POSIX.

Why? Because, even with ext3, the only way to correctly implement "replace file" on POSIX is to write the replacement file on the same directory, fsync() the file, close the file descriptor, fsync() the directory, and rename the file.

At that point, the whys have shifted the blame through userland programmers, POSIX fopen(), ext4, and back to userland programmers. Depending on where you stop, the solution can be:

1. fix all userland programs,

2. change ext4 to force sync upon closing a truncated file's descriptor (which got implemented),

3. upgrade the POSIX standard to have a dedicated "replace file" function call.

The truth is, userland is buggy, and the POSIX standard is arcane, so they both share blame. (But ext4, which was correctly implemented, is the only thing that got patched.)

http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocat...

Re:The Problem with ‘5 Whys’ (2016) by grok2 on Thursday April 11, 2019 @12:56AM (#19631271)
To me it seems like ext4 was the problem here -- it tried to provide a feature that wasn't adequately supported by the underlying OS.
Re:The Problem with ‘5 Whys’ (2016) by espadrine on Thursday April 11, 2019 @05:09AM (#19632296)
Delayed allocation was adequately supported.

Implementing it in ext4 increased the risk of zeroing out a file from “having all files updated in the past 20ms be zeroed out” to “having all files updated in the past 4 seconds be zeroed out” in the event of a system crash.

Re:The Problem with ‘5 Whys’ (2016) by tedunangst on Wednesday April 10, 2019 @01:41PM (#19627025)
Team two promised us this service would never fail, so we didn't check for errors...
Re:The Problem with ‘5 Whys’ (2016) by ben509 on Wednesday April 10, 2019 @06:03PM (#19629220)
The root cause may be something outside your team, but that doesn't mean you're saying you'll sit on your hands. "The service we depend on failed" can still mean, "and we need to implement better retries or redundancy" as a remedy.
Re:The Problem with ‘5 Whys’ (2016) by hinkley on Wednesday April 10, 2019 @03:32PM (#19627994)
So you know how in Refactoring you're supposed to be doing bottom-up design, but when the problem gets big it turns into top-down design masquerading as bottom-up? (aside: the Mikado method allows you to fix this)

Quite often when I'm doing 5 Why's I find myself socially engineering Why 3-5 to lead us to an action item we can do something about. There are any number of Why #5's someone can generate but a lot of them do not represent progress the team can get behind.

The fact that this happens with so many different teams and managers makes me wonder if it's me or there's just an obvious pattern of misuse going on with 5 Whys.

Re:The Problem with ‘5 Whys’ (2016) by dustindiamond on Wednesday April 10, 2019 @12:55PM (#19626558)
Kinda like ‘doing agile’ can mean anything, it seems.
Re:The Problem with ‘5 Whys’ (2016) by thnukid on Wednesday April 10, 2019 @01:02PM (#19626635)
Kinda like `Object Oriented` Programming, which is not about Objects.
Re:The Problem with ‘5 Whys’ (2016) by booleandilemma on Wednesday April 10, 2019 @01:07PM (#19626685)
OOP has some very clearly defined concepts, object being one of them, and the concepts are almost always baked into the languages themselves.
Re:The Problem with ‘5 Whys’ (2016) by danso on Wednesday April 10, 2019 @01:55PM (#19627157)
It's "Object Oriented Programming" not "Object Only Programming
Re:The Problem with ‘5 Whys’ (2016) by kochikame on Wednesday April 10, 2019 @08:25PM (#19629979)
> "5 Why" style analysis is really easy to steer toward whatever you want to blame.

This is the big issue. Ideology, bias and self-interest creep in and you can easily end up blaming whatever it is that you personally disagree with.

And that's to not even mention the "unknown unknowns", which by their very nature cannot be factored in to a "5 Whys" process, no matter their real world relevance to the problem at hand

Re:The Problem with ‘5 Whys’ (2016) by stcredzero on Wednesday April 10, 2019 @04:56PM (#19628713)
"5 Why" style analysis is really easy to steer toward whatever you want to blame.

How is this better than "Just So Stories?"

Re:The Problem with ‘5 Whys’ (2016) by kochikame on Wednesday April 10, 2019 @08:26PM (#19629983)
Exactly