News for hackers. Stuff that matters.
Re:What SRE Could Be by joshuamorton on Monday June 06, 2022 @10:12PM (#31649100)
The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned.

Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.

Re:What SRE Could Be by arminiusreturns on Monday June 06, 2022 @11:04PM (#31649415)
Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends.

This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.

Re:What SRE Could Be by donavanm on Monday June 06, 2022 @11:08PM (#31649435)
Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is

> A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.

So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?

Re:What SRE Could Be by joshuamorton on Monday June 06, 2022 @11:13PM (#31649464)
> Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is

No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that!

It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.

Re:What SRE Could Be by donavanm on Monday June 06, 2022 @11:38PM (#31649618)
> the SRE org is unilaterally able to stop supporting the product/implement a production freeze

Who exactly does this. And by what mechanism do you enforce:

> take the ability to deploy out of the product org's hands

If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent.

Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position.

Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in.

This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.

Re:What SRE Could Be by joshuamorton on Tuesday June 07, 2022 @12:14AM (#31649846)
> Who exactly does this. And by what mechanism do you enforce:

That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs.

> Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?"

Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this)

Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.

Re:What SRE Could Be by aluminussoma on Tuesday June 07, 2022 @12:49AM (#31650008)
> They are given the power to stop deployment. If they can't, they're not SREs

Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!

Re:What SRE Could Be by xorcist on Tuesday June 07, 2022 @07:20AM (#31652519)
Given the availability of Facebook, Netflix and the rest, one might think that Google is on to something there.
Re:What SRE Could Be by donavanm on Tuesday June 07, 2022 @10:57AM (#31655300)
> They are given the power to stop deployment.

Sweet. Given by whom

> Alice can tell Bob "no more deployments" is literally the point.

Fantastic. Why can she say & enforce this.

You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.”

To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.

Re:What SRE Could Be by joshuamorton on Tuesday June 07, 2022 @12:28PM (#31656724)
> Sweet. Given by whom

Directly, the answer is the permissions system. In my neck of the woods, devs are quite literally incapable of pushing things to production on their own. The automated tooling handles it 99% of the time, but in the case that they turn it off, only the SREs have production access necessary to actually push new things.

But I think you're speaking organizationally, and the answer there is that that agreement was made by SVPs and is enforced by the PRR process. When you adopt SRE support, you hand over the keys, so to speak.

> Fantastic. Why can she say & enforce this.

Because there's nothing Bob can do except escalate, and SRE leadership will support Alice all the way up to the SVP level.

> To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm.

My point is that the process creates the cultural norm, and enforces everywhere. That is, your org might have these good processes, and that's laudable. But if Andy turned around tomorrow and said "hey, this year your first priority needs to be shipping new features, because we're falling behind", would your org maintain that commitment? Would every exec? The same question goes for every exec speaking to their subordinates.

Its much more difficult to turn to the guy whose job is reliability and reports directly to the CEO and say "hey, your focus this year is shipping features". Like maybe you could say "your focus is shipping features reliably this year", and that should take the focus off of some other things that the SRE org is involved in, but I'm not convinced.

Or briefly, your approach is betting on there being a cohesive company culture of reliability that permeates every team, org, and manager. We both know that's not true, at Amazon or Google or any other company. They're all big enough to have different cultures in different parts. A distinct org and reporting structure, and split responsibilities, ensures that leadership has to be really explicit about such things ("lower your SLOs"), instead of just quietly deprioritizing enforcing or reporting on them. (And I'm generally a fan of multiple individuals with well scoped and distinct responsibility who are charged with reaching consensus over a single individual who makes a decision).

Re:What SRE Could Be by __turbobrew__ on Monday June 06, 2022 @11:25PM (#31649533)
Isn’t reliability a feature of the product? I am currently building out a team for a new product and reliability is quite high on our feature list. We have a embedded SREs who are working on improving our product reliability full time. We share the operational workload between the SWEs and SREs instead of just throwing all the operational problems at the SREs. This gives developers an appreciation for reliability and stability and makes sure that the SREs actually have the time and resources to improve the product instead of being a help desk.

It sounds like you needed a separate chain of command for SREs because your product leadership was incompetent.

Re:What SRE Could Be by dilyevsky on Monday June 06, 2022 @11:36PM (#31649601)
If your stack is complex enough and/or team is large enough the reliability work can be smeared so thin or be so intricate that it becomes nobody’s responsibility. Hence separate command chain that Google and some others use
Re:What SRE Could Be by solatic on Tuesday June 07, 2022 @01:06AM (#31650090)
> incentives are misaligned

If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to.

The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.

Re:What SRE Could Be by dilyevsky on Tuesday June 07, 2022 @04:43AM (#31651462)
My experience suggests that you can’t bolt reliability on the side when you suddenly decide to prioritize it. Not after certain scale at least.
Re:What SRE Could Be by solatic on Wednesday June 08, 2022 @01:21AM (#31663536)
Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
Re:What SRE Could Be by dilyevsky on Wednesday June 08, 2022 @10:27PM (#31676984)
This scenario seems overly naive unless there exists totally separate sre chain like I described up the thread.