On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis.
This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different.
It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.
The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time.
I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked.
There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.
Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.
Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”
1) Make a goal: we should get paged for at most N incidents per week. That goal should be closer to 0 than 10 IMO.
2) Track stats on this goal, both in aggregate and broken down by cases you think you can address separately. Example: tickets from alarms vs tickets filed by people. Alarms having to do with external dependencies vs alarms caused by your own bugs. Don't just make this a "it seems like we've had fewer pages lately" thing. Real numbers.
3) Review these stats on a graph every week. Someone should have an explanation of why they have spiked, why they haven't dropped, the breakdown of problem type, etc. There should be congratulations when they drop and a request for plans when they don't.
4) have management that can communicate upwards to leadership that ops improvement is a priority for your team and that you ultimately won't be able to continue other feature development if you are always mired in ops pain and people are either busy with mundanity or driven to leave the team.
5) dedicate time in each sprint to working on the most recent identified target from plans made in #3.
edit: formatted so indented list is readable on mobile. I have no idea how to do this without a code block. HN, please make this easier :)
One reality is more like "You are handed a system with a heavy on-call load. Make it better."
The other is "The system you built was great with 1e9 load, but now we're heading for 1e12 load. Make it better."
My short summary is, fixing on-call is a HUMAN problem, not a technical engineering problem.
Here’s an excellent place to start:
https://monitoring.love/articles/how-to-improve-on-call/
There is a wonderful list of talks and resources there, and those will lead you to yet more concepts and ideas to research.
Unless you have a leadership that recognizes the time sink on-call generates, all the best practices in the world are not going to help. This goes back to leadership culture. If they aren't in the trenches taking the same rotations with the on-call staff and at least simply staying up with them when the calls come in, then they will need metrics they believe to understand the deleterious effects of excessive on-call. This doesn't even begin to touch upon excessive on-call environments strongly signalling ignored technical and/or product debt that drags down new feature implementation and innovation. Leadership that truly comprehends this will allocate significant (20-25%) of time to addressing it, and push back on feature demands.
If you get to that point, then 80% of your struggle is over. Once you have leadership backing bringing the necessary time and resources to bear, reducing on-call is mostly lots of bookkeeping and documentation to identify trends and commonalities to prioritize and fix. A lot of the specific implementation details depend upon what you have available to leverage; ideally put someone in the PM role of tracking details for weekly stand-ups who is a relentless detailed note-taker and follow-upper.
I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things.
If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.