I've heard managers often tout that "if your code is stable you'll never get paged". But that hides the fact that you're still on the hook to be available. So either you start shirking your responsibility or you reshape your life around some percent of non-working hours being owned by the company still.
That being said: the SRE model has encouraged orgs to improve observability and set clear expectations and it often coaxes teams into building more reliable systems under the threat of limitless OC toil.
During that time I keep my laptop and my phone that can tether with me. I've been on call and at amusement parks. I got paged once and had to go out to the car and work an hour. Calculated risk on my part. What I should have done is ask a team mate if they would cover me for the several hours I was at the park.
However, I do want to work in places where I am responsible for operating what I build. I want to know how well the service I built is doing. If I promise it will be up 99.99% of the time but is not, then I need to know if its down so I can figured out why it went down.
Being on call for systems you build also leads to better software. It makes me design software in ways that are helpful to investigate when it fails. e.g. my error messages provide a lot of context. There is investment in making sure the service is able to terminate gracefully. Metrics are instrumented to locate when things might be off.
What this means is that for things that I build, I often don't get paged as often. Its in my interest to ensure that the service is honest, well designed and well built. When I do get paged, its for something that's seriously wrong.
Anyways, I've been lucky to be in this position. What I've observed for many teams is that they inherit flaky systems which they have to make more reliable; but in the meanwhile every OC shift is an absolute slog, requiring 40 hours/week dedication by people.
If I'm going to be responsible for what I build at all times, then I'm going to be compensated for what it's earning at all times.
That means options, stock, or profit sharing (with a large preference for stock).
Otherwise I feel like this relationship is strictly abusive.
Alternatively, my contracting rate is 150 an hour. I'll agree to work up to 70 hours a week, but everything after 50 is compensated hourly. On call counts, regardless of whether I actually get pinged.
On-call is the type of egregious abuse of employees that an IT union would be able to fight against. Right now companies can take advantage of people who are desperate (H1-B, young parents, etc) and don’t have the freedom to take a stand like the above poster but if all IT workers banded together it would be possible to fight against the practice.
Expecting another group to maintain your code seems an awful lot like throwing it over the wall.
You should have faith in the systems that you build. If you don't want to carry a pager for them, why does someone else want to do it?
SREs have global responsibility to the system and have a correspondingly global view. If your system broke down because of a bug you wrote, that’s on you. If a system three times removed from yours that you didn’t know depended on you broke because you changed a non-API behavior, that’s where SREs shine: they know how to quickly isolate the problem, roll back the necessary systems, and define how to avoid similar problems in the future.
Only ever seen it possible to turn around with empowered engineers that have the cycles to fix crippling systemic defects and the business has the means to support it. Otherwise it’s basically a Sisyphean task to maintain things and the business should have offshored the labor by then to save on costs if the cost-feature curve isn’t working out in growth stages.
It also assumes that designing production infrastructure or even systems level thinking is a skill that most software developers have. Certainly, enough do that it's common, but it's not universal.
In this particular case, the org standardized on terraform. Some developers welcomed the ability to self-service. Others wanted nothing to do with it.
Microservices merely ensure that the complexity of using different tools per microservice does not lead to an increase of maintenance burden.
Nonetheless, you still have a maintenance burden if every microservice is built upon different tools and processes, since you cannot address patterns of problems easily that occur within combinations of patterns and processes. The cardinality just increases with every tool and process added into the mix.
This burden of maintenance automatically takes its toll on productivity. Teams will not be able to create, maintain or iterate on code that produces business value (as opposed to managing the platform of microservices itself)
I think there is value in a well defined set of processes and tools because there will eventually be platform concerns that become increasingly difficult otherwise.
I do not have insight in how most companies with successful microservice architectures achieved their success, but I would bet my life that a majority of those companies do not let their engineering teams use tools and processes arbitrarily, unless it serves to REALLY produce value that would be unachievable by the tools and processes used up until that point (for example, because performance is usually sufficient, but a specific service suddenly needs to outperform everything that came so far, so you use C, Go or Rust instead of Python)
I am biased into thinking this because I suspect that most of these companies did either start with monolithic architecture (i.e. supporting multiple languages was impossible or just not feasible enough) or they started with a microservice approach focusing on producing value as quickly as possible after an initial ramp up time. Supporting many different tools and processes from the get go would make the ramp up time longer than necessary.
TL;DR Microservice platform maintenance suffers if tools and processes are chosen freely for each microservice, without any push towards unification.
Oof I need to sleep. Don't mind me, just trying to sort myself out.
The crucial feature of microservices is to enable large number of people to work on delivering the same service, but not being constrained by the traditional modes of software release, where the entire monolith could only be updated at a fixed cadence and where the whole thing had to pass through a rigorous suite of tests etc.
One freedom that this gives a team is to choose a different language for their service... on this point there is large consensus. But one need not stop there; you could technically run your service on another cloud with a completely different deployment architecture. All that is asked for this insane freedom is that your service honor the SLA's that you provide. SLA's define the interface, everything else is left to the team.
That being said, shared tooling does provide some nice benefits where learning is shared across teams allowing new ones to bootstrap quickly. However, that is something the team should be allowed to choose.
Commonly we see a team of 20 maintaining 40 microservices, which becomes very difficult to manage and provides little gains. IMHO, this si the wrong approach. Microservices are tiny from the point of view of Netflix, or Amazon.
I implemented a similar change at a previous company. Prior to the change, the operations team fronted all on-call issues. After, anything related to the software/services was handled by software teams. We went from nightly alerts to less than monthly. As a bonus, time to diagnose and repair went waayyyyy down due to much better logging.
Note, better logging <> more logging in this case. It meant less spurious errors/warnings, more useful INFO, and better observability of the system.
During the old days of “you commit code and ops runs it” and during transition to “you build and run it”, we did have some bonuses for being on-call or doing stints in problem management. Now, we bake that expectation into base salary except in locales where we have to split it out to comply with local laws.
Along the way, we've given enough merit increases, market adjustments, and promotions that it's impossible to say whether we had a bump specifically because of the industry's new philosophy on ops. If you wanted to make the case that we did, you could. If you wanted to make the case that we didn't, you equally could.
The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night. Fast forward and now I can count on one hand the amount of pages I've had on my last multiple on call rotations. Work is prioritized for system reliability because we feel the pain.