News for hackers. Stuff that matters.
Re:What SRE Could Be by aluminussoma on Monday June 06, 2022 @09:05PM (#31648609)
SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering.

Pay is a rough signal on what a company expects from SRE. If SREs get paid as much or more than Software Engineers, it is a good sign. Yet even that signal is not accurate. I know highly paid SREs stuck in Ops Hell. You could not pay me enough money to do a job like that. It is also not sustainable because eventually the company will fire you and replace you with a lower paid employee who can do that job just as well.

I strongly oppose organizations that have SRE separate from Software Engineers. I strongly believe that SREs must be fully integrated into Software Engineering teams (aka the embedded model), with the same reporting lines as their product colleagues.

Keeping SRE organizationally separate has done a great disservice to SRE. SRE 2.0 needs to tear down the mini empires that SRE 1.0 built and get back to software engineering.

Re:What SRE Could Be by dijit on Monday June 06, 2022 @09:34PM (#31648823)
> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering.

SRE is Sysadmins but with better contracts between teams, and automation focused.

I haven't had much trouble with that definition, does away with all the cruft of the "Production" talk by Benjamin Treynor Sloss and the majority of the SRE books (which are a prescription of a method of creating those contracts).

That said: "DevOps" and Systems Administration in general has had an identity crisis for a while. People don't like the idea of "Sysadmin" for some unknown reason, but give the same role to someone and call them "DevOps" and suddenly all is well.

Same is now true of SRE.

Someone on HN said to me: "DevOps is how you get Engineering to respect Systems Administration".. Maybe the same is true of SRE?

I definitely feel the identity crisis of DevOps is much, much worse than that of SRE- but I guess that depends on your circles.

Re:What SRE Could Be by kodah on Monday June 06, 2022 @10:57PM (#31649370)
> SRE is Sysadmins but with better contracts between teams, and automation focused.

That's not even close to true, which I think is OPs point. For instance, I have an extensive SRE-SE background, but I'm also a software engineer. If I was tasked with "sysadmin" stuff, or shepherding peoples Jenkins jobs every day I'd probably lose my mind.

Re:What SRE Could Be by dijit on Monday June 06, 2022 @11:24PM (#31649526)
I think we misunderstand each other completely.

There are many types of systems administrators; when we talk about as SRE in particular there some overlap with what I was doing as a “Linux Systems Administrator” in a Platform Engineering team, in 2009.

Automating migrations, distributed job control, instrumenting the message queue system, managing cache invalidation across our entire fleet of memcache servers (which was ugly hacked with MySQL, not proud), as well as “maintenance” programs to upgrade “versions” (multi-tenant SaaS) and warm caches, dark launches, canary deployments… well you get the picture.

Of course I was responsible for the reliability of the systems these things ran on, but there was a really-really strong focus on automation as part of the job, and this was when puppet was quite new, we were stuck with tools like cfengine.

Coddling Jenkins sounds like build or release engineering. Which is also a separate thing.

People seem to forget that systems administration has a focus on software development too- I mean for goodness sake terraform, ansible and salt all come from sysadmins; I suspect it’s because of some very foolish admins proudly exclaiming they don’t code. But they were always the bottom rung and end up being “promoted” to management at some point.

Re:What SRE Could Be by kodah on Tuesday June 07, 2022 @01:54AM (#31650349)
I get more of what you're saying now, but I still think that reinforces OPs point because, to me, what you described is a Systems Engineer not a Systems Administrator. OP is asking for some structuring as to what these roles require in terms of knowledge. To me, "Administrator", "Engineer", "Technician", etc all prescribe different degrees of knowledge and also usually correlate with responsibility.

As for "SRE" vs "DevOps" vs "Systems Engineer", I'm not so sure. For instance, I typically expect SREs to be able to write application software and orchestration software. I have similar expectations of someone who uses the title Systems Engineer but I might expect less application software development experience. DevOps I think I'd rather forget is a word.

Re:What SRE Could Be by xorcist on Tuesday June 07, 2022 @07:11AM (#31652444)
Systems engineer is a title that seems to have fallen somewhat out of favor, but it wasn't unusual that it was a less hands-on role than sysadmin/SRE. Sometimes close to a system integrator but could also be what is now known as an infrastructure architect.

Who do you think did the SRE work before that name was invented? Most likely it was your friendly sysadmin. It is the type of role that is not much noticed when done well, and probably that's why some feel it underappreciated enough to keep inventing new job titles. It's still the same job, pretty much the role that requires privileged access.

Re:What SRE Could Be by kodah on Tuesday June 07, 2022 @01:24PM (#31657500)
I was raised in the South and grew up across the midwest, so that may color my perspective.

The place where I encountered sysadmins the most was people who managed a small collection of assets; the keyword being managed. I didn't see them inventing entirely new systems. Working across DCs, I would see Systems Engineers building these new systems, being escalation support for them, and at times coding up entirely new infrastructure applications.

I don't really buy the idea that administrators are underappreciated, I just don't think it accurately captures what a systems person on a product team actually does, much less the distinction between a SRE-SE and a systems person.

My base argument is that it's not the same job at all. A good Systems Engineer does not guarantee a good SRE-SE because the job is that much different. One of the notable distinctions is writing application code, which is much different from Terraform, Ansible, Salt etc

Re:What SRE Could Be by apple4ever on Friday June 10, 2022 @09:03PM (#31701010)
I have to disagree. SRE=SE=SA. What they exactly do depends on the job. Some can write application code, some cannot. But they are all the same thing. I've worked at SRE jobs where there was no application code writing, and I've worked at SE jobs where there was application code writing.
Re:What SRE Could Be by Rapzid on Monday June 06, 2022 @11:17PM (#31649483)
Same, background in cloud/systems engineering, SRE, and staff full stack engineer. Or, FullStack++ as I sometimes like to call it haha.
Re:What SRE Could Be by pojzon on Tuesday June 07, 2022 @03:17AM (#31650855)
How do you call an Engineer that can do everything beside Frontend ?
Re:What SRE Could Be by nunez on Wednesday June 08, 2022 @12:15AM (#31663116)
Reliability Engineering is much wider than the scope of a sysadmin. SREs are software engineers whose focus on the reliability of systems and platforms. Could be sysadmins; could be traditional SWEs; knowledge of both is required.
Re:What SRE Could Be by apple4ever on Friday June 10, 2022 @09:04PM (#31701023)
No they are the same thing. Reliability Engineering was always part of a sysadmin's job, but how much depended on the job.
Re:What SRE Could Be by daenz on Monday June 06, 2022 @10:12PM (#31649102)
Former Senior SRE here. Was doing everything: working with engineering teams to build a microservice authoring framework. Putting out fires and responding to outages. Adding monitoring/metrics/logging to all the things. Building productivity tools for engineers. Spinning up new software in sandboxed environments for evaluation. Onboarding and offboarding new engineers. Coordinating downtime for software upgrades. Refactoring our infra and making it more IaC. Security audits.

Our team was basically a dumping ground for anything that required experience in many different types systems and/or privileged access. On top of that, we had projects were shoe-horned into roadmaps. So while 90% of our duties were reactive to what the org needed, we also had the pressure of trying to meet deadlines for normal software engineering work. It was unhealthy and I wanted very much to not be defined as an "SRE"

Re:What SRE Could Be by samstave on Tuesday June 07, 2022 @04:45AM (#31651469)
Director of Ops title for a decade here:

Add on top of this managing multiples of you and having to do the CFOs work for the CEO and litterally manage to the penny the instances we spun up and dealing with all the expectations of ops WRT to developing tooling on top of tooling.

Re:What SRE Could Be by hiddenwaffle on Monday June 06, 2022 @10:30PM (#31649220)
My company expects SRE to be synonymous with cloud administrator, acting as the gateway to doing anything AWS-related.

They've also asked us to troubleshoot for database backup processes, write Crystal Reports on user activity, and be Slack admins. Some of the developers believe that we do manual production deployments of their builds (we don't). I get requests all the time from users asking if I can make accounts on various systems for them.

I was given a programming test for this job :(

Re:What SRE Could Be by dub on Monday June 06, 2022 @10:49PM (#31649329)
Agreed about having SRE fully integrated into feature teams in terms of day-to-day work, seating charts, team building events, etc.

From a management perspective I think it still makes sense for SRE to have a different reporting structure. Feature teams generally aren't rewarded for investing in long-term code and production health. If you report to a feature team manager there will often be downward pressure to focus less on production health and more on helping features ship quickly.

Giving SRE a degree of independent oversight serves as a system of checks and balances against those feature team pressures.

Re:What SRE Could Be by throwaway894345 on Monday June 06, 2022 @09:39PM (#31648878)
One of the functions that SRE provides in our organization is building out the platform and solving cross-cutting problems. How do you do that without some reporting structure? Shared ownership and guilds can only go so far since each team has its own priorities and incentives (and will rarely choose to yield engineering time to platform things if they can devote it to product things).

I’m not arguing against an embedded model, but I do think there’s tension that’s not easily resolved.

Re:What SRE Could Be by aluminussoma on Monday June 06, 2022 @09:54PM (#31648986)
In my division, there is an infrastructure team and product teams. Those platform type projects are handled by the infrastructure team. In a product only team, it will always be difficult to work on such projects.

Ultimately, the answer ties back into the identity question: what is SRE?

Re:What SRE Could Be by throwaway894345 on Tuesday June 07, 2022 @09:15AM (#31653768)
In our org, we have SRE who handle platform and SRE who are embedded on product teams, but all report up through the SRE org. One of the goals is to spread more infrastructure knowledge to the engineering teams to reduce their dependence on SRE so that SRE is mostly doing platform work and special infra-intense projects.

I’m not saying this is the one true way or even a particularly good way, and I agree about the identity crisis (same as with DevOps half a decade ago).

Re:What SRE Could Be by SassyGrapefruit on Tuesday June 07, 2022 @07:33AM (#31652624)
We use an "embedded model" with separate teams. I run an SRE organization with 10 individuals. At any given moment 3-4 are embedded on teams pursuing some objective. The rest are doing incident response and SRE specific projects from the big pile of sprint work.

We did this because in our case most development problems didn't have a reliability dimension. We build mathematical models. Do I need an SRE involved in the discussion of switching from one statistical approach to another? In most cases no. But when that team deems themselves limited by their current walled garden and want to explore/operationalize new technology then we need an embedded SRE to facilitate that.

Re:What SRE Could Be by kodah on Monday June 06, 2022 @10:56PM (#31649365)
> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering.

SWE has an identity crisis. It means too many things to too many people.

Just kidding, our entire industry is filled with titles that mean nothing. I agree, we should fix that.

Re:What SRE Could Be by dilyevsky on Monday June 06, 2022 @09:28PM (#31648776)
Imo embedded mode is no better - usually just a token ops person that everyone on the team throws over the fence to.

I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.

Re:What SRE Could Be by pramodbiligiri on Tuesday June 07, 2022 @12:01AM (#31649755)
A recent talk I saw suggested that SREs report into the COO (Chief Operations Officer) and not the CTO or VP of Engg. I thought that’s a good idea.

The COO is interested in the stability of ongoing operations whereas the other two roles are about launching new things (Unless I have gotten the definition of COO wrong).

Re:What SRE Could Be by nunez on Wednesday June 08, 2022 @12:17AM (#31663131)
organizationally that's a bad idea at a lot of places because the CTO will usually have more oversight and a bigger budget to match, and SREs are closer to SWEs than traditional ops
Re:What SRE Could Be by aluminussoma on Monday June 06, 2022 @10:05PM (#31649063)
In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot.

They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs).

SRE 2.0 needs to make the embedded model work.

Re:What SRE Could Be by joshuamorton on Monday June 06, 2022 @10:12PM (#31649100)
The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned.

Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.

Re:What SRE Could Be by arminiusreturns on Monday June 06, 2022 @11:04PM (#31649415)
Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends.

This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.

Re:What SRE Could Be by donavanm on Monday June 06, 2022 @11:08PM (#31649435)
Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is

> A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.

So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?

Re:What SRE Could Be by joshuamorton on Monday June 06, 2022 @11:13PM (#31649464)
> Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is

No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that!

It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.

Re:What SRE Could Be by donavanm on Monday June 06, 2022 @11:38PM (#31649618)
> the SRE org is unilaterally able to stop supporting the product/implement a production freeze

Who exactly does this. And by what mechanism do you enforce:

> take the ability to deploy out of the product org's hands

If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent.

Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position.

Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in.

This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.

Re:What SRE Could Be by joshuamorton on Tuesday June 07, 2022 @12:14AM (#31649846)
> Who exactly does this. And by what mechanism do you enforce:

That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs.

> Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?"

Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this)

Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.

Re:What SRE Could Be by aluminussoma on Tuesday June 07, 2022 @12:49AM (#31650008)
> They are given the power to stop deployment. If they can't, they're not SREs

Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!

Re:What SRE Could Be by xorcist on Tuesday June 07, 2022 @07:20AM (#31652519)
Given the availability of Facebook, Netflix and the rest, one might think that Google is on to something there.
Re:What SRE Could Be by donavanm on Tuesday June 07, 2022 @10:57AM (#31655300)
> They are given the power to stop deployment.

Sweet. Given by whom

> Alice can tell Bob "no more deployments" is literally the point.

Fantastic. Why can she say & enforce this.

You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.”

To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.

Re:What SRE Could Be by joshuamorton on Tuesday June 07, 2022 @12:28PM (#31656724)
> Sweet. Given by whom

Directly, the answer is the permissions system. In my neck of the woods, devs are quite literally incapable of pushing things to production on their own. The automated tooling handles it 99% of the time, but in the case that they turn it off, only the SREs have production access necessary to actually push new things.

But I think you're speaking organizationally, and the answer there is that that agreement was made by SVPs and is enforced by the PRR process. When you adopt SRE support, you hand over the keys, so to speak.

> Fantastic. Why can she say & enforce this.

Because there's nothing Bob can do except escalate, and SRE leadership will support Alice all the way up to the SVP level.

> To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm.

My point is that the process creates the cultural norm, and enforces everywhere. That is, your org might have these good processes, and that's laudable. But if Andy turned around tomorrow and said "hey, this year your first priority needs to be shipping new features, because we're falling behind", would your org maintain that commitment? Would every exec? The same question goes for every exec speaking to their subordinates.

Its much more difficult to turn to the guy whose job is reliability and reports directly to the CEO and say "hey, your focus this year is shipping features". Like maybe you could say "your focus is shipping features reliably this year", and that should take the focus off of some other things that the SRE org is involved in, but I'm not convinced.

Or briefly, your approach is betting on there being a cohesive company culture of reliability that permeates every team, org, and manager. We both know that's not true, at Amazon or Google or any other company. They're all big enough to have different cultures in different parts. A distinct org and reporting structure, and split responsibilities, ensures that leadership has to be really explicit about such things ("lower your SLOs"), instead of just quietly deprioritizing enforcing or reporting on them. (And I'm generally a fan of multiple individuals with well scoped and distinct responsibility who are charged with reaching consensus over a single individual who makes a decision).

Re:What SRE Could Be by __turbobrew__ on Monday June 06, 2022 @11:25PM (#31649533)
Isn’t reliability a feature of the product? I am currently building out a team for a new product and reliability is quite high on our feature list. We have a embedded SREs who are working on improving our product reliability full time. We share the operational workload between the SWEs and SREs instead of just throwing all the operational problems at the SREs. This gives developers an appreciation for reliability and stability and makes sure that the SREs actually have the time and resources to improve the product instead of being a help desk.

It sounds like you needed a separate chain of command for SREs because your product leadership was incompetent.

Re:What SRE Could Be by dilyevsky on Monday June 06, 2022 @11:36PM (#31649601)
If your stack is complex enough and/or team is large enough the reliability work can be smeared so thin or be so intricate that it becomes nobody’s responsibility. Hence separate command chain that Google and some others use
Re:What SRE Could Be by solatic on Tuesday June 07, 2022 @01:06AM (#31650090)
> incentives are misaligned

If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to.

The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.

Re:What SRE Could Be by dilyevsky on Tuesday June 07, 2022 @04:43AM (#31651462)
My experience suggests that you can’t bolt reliability on the side when you suddenly decide to prioritize it. Not after certain scale at least.
Re:What SRE Could Be by solatic on Wednesday June 08, 2022 @01:21AM (#31663536)
Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
Re:What SRE Could Be by dilyevsky on Wednesday June 08, 2022 @10:27PM (#31676984)
This scenario seems overly naive unless there exists totally separate sre chain like I described up the thread.
Re:What SRE Could Be by dilyevsky on Monday June 06, 2022 @10:56PM (#31649362)
Yes SLOs are kind of like OKRs of sre management - they only ever as good as the objectives and objectives are often not set very well by senior leadership. Ultimately that stems from the fact that reliability is not valued by the market. You see a lot of the same things in security (outside of the boneheaded compliance cya).
Re:What SRE Could Be by samstave on Tuesday June 07, 2022 @04:54AM (#31651543)
I feel like you haven't heard of BOFH.

Bastard Operator From Hell -- A famous IT trope. Which was typically a sysad for a mainframe/Mini-Computer... Usually on a University Campus....

Think Lazlo from Wierd Science, but an asshole.

I actually worked with a BOFH. (Sadly he had a heart attack on the CalTrain Station in Mountain View, and died.)

He was the AS/400 admin for a place where we manufactured all of SUNs Software (meaning made the SUNos CDs manuals packaging etc) and this guy was a BOFH

Sales folk would call up to the helpdesk and request a password reset.

This one time, he set the users password to "STUPID!" ...

I fired him for that.

He had a heart attack shortly thereafter....

:-(

Re:What SRE Could Be by brundolf on Monday June 06, 2022 @11:21PM (#31649507)
> I strongly oppose organizations that have SRE separate from Software Engineers

Can confirm, we have an SRE team that's totally separate from engineering and it's a huge and constant bottleneck. We've literally re-architected parts of the application to avoid having to go through SRE for things.