throwaway210620

> the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response

Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.

In my experience:

  1) Make a goal:  we should get paged for at most N incidents per week.  That goal should be closer to 0 than 10 IMO.
  2) Track stats on this goal, both in aggregate and broken down by cases you think you can address separately.  Example: tickets from alarms vs tickets filed by people.  Alarms having to do with external dependencies vs alarms caused by your own bugs.  Don't just make this a "it seems like we've had fewer pages lately" thing.  Real numbers.
  3) Review these stats on a graph every week. Someone should have an explanation of why they have spiked, why they haven't dropped, the breakdown of problem type, etc.  There should be congratulations when they drop and a request for plans when they don't.
  4) have management that can communicate upwards to leadership that ops improvement is a priority for your team and that you ultimately won't be able to continue other feature development if you are always mired in ops pain and people are either busy with mundanity or driven to leave the team.
  5) dedicate time in each sprint to working on the most recent identified target from plans made in #3.

This isn't particularly complicated, and typing it out almost sounds like I'm giving you worthless common sense advice, but I think the key here is that multiple levels of your organization need to commit to making this important enough to spend time reviewing it, agree it is a priority, and put actual dedicated dev work into it.

edit: formatted so indented list is readable on mobile. I have no idea how to do this without a code block. HN, please make this easier :)

The lack of investment into decreasing on-call pain is a real factor. I work on Oracle cloud (OCI) and at least some of the orgs (VP-down) have figured out that this is something worth focusing on, and the on-call gets better and better as a result. My original team had an average of something like 50 pager-worthy (sev2) events per week until we got moved into a new org that had the right philosophy and we relentlessly drove that down because management realized that engineers made miserable by mundane ops fake-emergencies would eventually get fed up and leave, and that's not what they wanted (afaik, OCI has no such forced attrition). So we got put on a program of relentlessly tracking and categorizing the sev2 counts and committing to improving those numbers over a period of time. 25% of dev sprints were dedicated to improving ops (tools, better alarms, fixing long-backlogged bugs that led to pages), and now that team's ops are pretty easy and they are free to work on new features, which everyone prefers. I've since moved to another team whose ops had already had this optimization done, and I've never experienced a bad week of on call there.

I won't pretend OCI is a panacea (lol google oracle cloud toxic work environment for latest stories) but at least they don't lack this particular piece of wisdom. The sheer number of regions they plan to operate doesn't really allow them to ignore dumb ops problems.