oncallhacks

In one of my first jobs at Fortune 20 company, they had on-call escalation procedure, longer you take to fix an issue, more and more people get called. It was something like on-call developer > team lead > whole team and manager > director > senior director > VP... Usually, after an hour, we had to get manager and whole team online. Manager would be freaking out and driving us hard to fix outage with whatever means necessary and not let it escalate further. They would make it sound like our whole team will suffer if director got on the call. Of course, they said that we can create issues and fix them and of course, there were always more important issues.

My team quickly and informally learned if you want to take care of technical debt, let the higher ups feel the pain. Even when nginx/some process died and just need to restart, we would wait and really try to let those calls escalate to director. And anytime director had to join bridge, those issues would become highest priorities and fixed permanently.

It also became a game where manager would keep asking to do simple things to fix and we would be stalling or looking at unimportant things. Pretending like slow internet connection, dropped calls, locking ourselves out from the box, etc.

And in our team we were all PS3 gamers, so we would get on GTA4, and start having fun. A few times we even yelled at each other for messing up in the game while unmuted.

These days depending on company's culture I may fix issue quickly or I may let it escalate to right level where we will get support for fixing the issue for good.