Re:Proximate vs. Root Causes: Why You Should Keep Digging to...
by contingencies on Sunday September 22, 2019 @10:56PM
(#21045596)
Agreed, after identifying contributing factors or manageable interfaces process-oriented fixes are most powerful. That's why CI/CD is so awesome, RCS/VCS pre-commit and post-commit hooks allow you to automate the checklists you need to solve entire categories of issues.
These will be different depending on your perspective/level of power. For a SWE, it will be something like, "If we had monitoring of this variable, we would have gotten notified faster." Or, "If we had this type of testing, we'd prevent this class of errors." For a VP, it will look more like, "If our incentive structure were changed this way, or if our dev infra team built this kind of tooling, my SWEs would be better enabled to prevent this outage." Often there will be more than one point of leverage, but you only have so much time in your life, so the question is which one is most important or likely to be effective.
Examples of points of non-leverage are, "Joe submitted this busted CL." That is in the past and is not changeable. If Joe is in a habit of submitting busted CLs, he might need counseling, a PIP, or failing all else, to be fired. However, typically the submission of broken CLs is a systemic problem, not a case of a single sub-par engineer. You have to find the leverage that will allow you to change the system.