And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with.

Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about the pipelines, the kubernetes clusters, the containers, anything outside of their programming code basically.

So I still need to use all the communication skills I learned decades ago when I was doing end user support for people who didn't even work in IT.

It really feels worse than back in my websphere days, where in order to see app logs, we had to submit a ticket to the hosting teams and wait 2 days for a .log file to be sent back through email. I'm consistently left wondering if we've actually gained anything.

We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build.

Fix the cause, not the symptom.

Edit: And because I know it's coming... if IT security (or insert other org here) is the reason things can't be better, then it becomes the devops tools team's job to cajole them into doing it anyway, on behalf of their users.

No one realizes some dev teams can't see their own deployment logs. No one realizes teams don't have the correct Jenkins access. No one realizes there's a mysterious gatekeeper to use the corporate time series database as a service. No one realizes all new containers are transferred to a few people who have the rights to publish them.

And when I say "no one realizes", I mean two things. One, the people who do know, because they're the ones trying to do it, have given up bitching about it, because they did for years and nothing happened. Two, the people who can actually change things don't know, either because they actively ignored / forgot or because the requests never reached them (usually because a middle transport manager didn't understand what was actually being asked).

And so... the status quo prevails, and everyone has a vague sense that something isn't working, but doesn't know what to do about it.

And to that problem, the best solution I've found is to get the fixer and the problem user as close as possible, wherein the problem is usually revealed as a relatively simple change or feature oversight. And for that, discovery and user interviews are the most reliable method.

You mean put development and operations together, into some kind of dev..ops.. :)

I'm sorry for the snark, it was not meant as disagreement with what you said. The whole idea of the role was to try to remedy that exact issue, but old habits die hard and sometimes things just change names but stay the same. I shouldn't throw stones, I'm struggling at our shop to steer outside of that behavior as well.

The same thing with a single DBA or whatever admin team. My dream team structure is a task-force like one. Each team has its own DevOps/DBA. They don't have to be dedicated roles, but someone in the team needs to be well-trained on those topics.

But I guess this model never scales in larger corporations so eventually they all adopt the One X-admin team model.

Looking back at the last twenty years, oscillating between central sysops and everyone doing their own thing (is that "devops" ?), I feel that an optimum requires both: local skills to iterate fast, central skills to consolidate. Striking a budgetary balance is difficult, but at least ensure that some local skills remain - sell it to central ops as a way to get the trivial demands off their backs !

It usually happens in practice ("Sarah knows someone in security"), but there's no reason "devops liason" can't be an explicit 2nd+ hat for someone on a team to wear.

Centralized teams are necessary for budgetary reasons: one person on every team asking for a devops product will never make a coherent case to management for why it's needed, but a team will.

But having been on both sides of the equation, the central-decentralized relationship usually breaks down because there's always a new face on the right side of that equation, and that requires reteaching / recoordinating all the basics. More consistency on the user side (a hat) helps significantly.

And ultimately, it's a trust problem: do I (central person) trust this person who's asking me for something? If they're new to me... probably not very much. If I've been working with them for awhile... probably a lot more.

Personally? Make a list of all the teams that use your product. Schedule interviews (30 minutes to an hour for initial) with a few that use it the most, and a few that use it less. Ask them what the best and worst parts are. Ask them details about the things they bring up, and prompt about related areas they might be struggling with too.

If you get any repeats from team to team, assume those are systemic issues and try and find a solution.

The hard part usually isn't figuring out the nuances, but rather realizing that someone wants to do something at all and/or can't because of some restriction. Assume you know nothing about how they actually do what they do and listen closely.

1. Find early adopters of the thing you're building who are willing to use and give feedback (bugs, friction logs) in the early stages of the feature.

2. Build relationships with your users / VIP users and meet regularly to discuss friction points and upcoming features (feasibility, etc). To make it time efficient, make sure you're talking to staff engineers who can speak on behalf of a significant portion of your user base. Make sure your team is closing the feedback loop with these engineers (i.e. shipping things that address their concerns) in a timely fashion.

Anyone here know a good legal, compliance or best practice reason for blocking all access, or did the infra/sec team just get carried away here?

IMO the biggest issue other engineers tend to have with security is that it needlessly restricts them to the point where their job sucks. It's just as important to avoid "too much restriction" as "too little restriction".

The philosophies include:

1. Applying a maxim like "Principle of Least Privilege", or "That which is not allowed, is forbidden". Maybe through choice, maybe due to standards like PCI-DSS and SOC2.

2. Blocking access isn't a problem, because we're going to grant access requests quickly. Just as soon as we fill these five vacancies on the team....

3. Surely bugs making it to production will be a very rare event? Aren't these developers testing properly?

4. Don't we have test environments for troubleshooting? Isn't the heart of devops that you have a test pipeline that you have confidence in? If your developers think they're likely to deploy buggy software to production, maybe you aren't ready for devops...

5. Any large organisation will, at some point in its history, have had some chump run their test code against the production system and send every customer an e-mail saying "Hello World". If that happened when you had 30 developers, I doubt hiring standards are any higher now you have 300 developers! Stopping that happening again is common sense professionalism.

6. Would you want Google's 100,000 engineers to have read-only access to your gmail account? Of course you wouldn't, you've got loads of private shit in there. And if you were CEO, would you want an intern on a 2 week placement to be able to download the entire customer database? Of course you wouldn't. People having access - even read-only access - to the production system is a bad thing.

7. Locking down a lot of things gives the security team more political power, as every request to reduce restrictions or expedite a request is a favour owed.

Coming from a team that builds and supports an end user suite of applications. I'm not asking you to bend over backwards like I would have to do for a client. But at least don't enter in to conversations with that attitude. You are in a service organization. You deliver value insofar as you can help the next link in the chain.

I swear to god the next guy who sends me the screenshot of an error log...

Still, there's a weird expectation that operations is a bunch of elite developers, knowledgeable about all programming language, who just choose to do operations because they enjoy getting calls at 3AM

The amount of engineering leaders that allow it to happen is far more frustrating, at least IMO it is.

And this particular developer is my main problem in that project. So it's not all developers who are bad, it's about the same percentage as regular end users I'd say.

UGH.

Pull up a seat at the bar with us.

I do think it’s unavoidable though. No company will have a truly open stack. There will be customized versions of everything, with caveats only the infrastructure team or developer-tools team are aware of. Naturally these teams become the go to folks to ask questions and resolve issues.

I don’t know if a good way around it. Documentation works to an extent. But at some point you have to decide what parts of the system you want to expose to developers, and whenever that happens you become the layer they communicate with rather than the underlying technology.

DC ops is in charge of the physical system, from racking and stacking to hardware replacements.

NetOps handles the networking stack, managing the routers and switches, vlans, announcements, IP block management, network access requests, etc.

SysOps handles provisioning systems, working with dc ops to get systems repaired, laying down the OS, managing the inventory system, and anything else the systems need to be ready to run our software.

Then DevOps is responsible for all the software that our company writes, and managing how developers get their software onto the machines, how they configure the software, and how they interact with their software in production.

The lines are obviously not that clean cut, but that is the basic idea.

Sometimes, automation and tooling are written by seperate teams, or are done by teams of different disciplines of operations/engineering.

I have almost never seen a network and system groups NOT being seperate. Mind you this is in DC/ISP's environment, where the complexity of networks is usually the mainstay or a heavy dependency of the business.

only it's not what the idea behind it was. It was exactly the opposite of another compartmentalization. it was supposed to bridge silos not carve out new job titles for quality control

It's like the article says:

> At some point in time, this idea has been so widely misunderstood that the wrong definition of DevOps became the right one.

See also: 'agile'.

I work as a sort of "dev tooling engineer" or "platform engineer" for several companies who are our customers.

After years of consulting in the area of DevOps, the pretty smart people I work with figured out that while the developers' user story in the process of devops transformation is a "revelation", the one thing that usually gets forgotten is the maintenance and upkeep of those tools. So we set up a pilot project where we started hosting, maintaining and upkeeping a set of tools consisting roughly of Jira+Confluence, Jenkins as the CI/CD server, Artifactory for binary hosting and Bitbucket/Gitlab for source code.

It has worked damn well for the first pilot company (signed on 5+ years ago), the second one, and by the third the platform started selling itself (getting a public reference from the first couple companies helped a lot). The second reason for the easy buy-in is to use known software that our customers could host themselves if they had to. This practically means that we aren’t vendor-locking them with our own proprietary tools. We just take care of off-the-shelf tools.

If there is a competent tooling team that takes care of the intricacies of the tools and maintains their uptimes, the dream of "you build it you run it" can still live. There is a giant gap between "you build it you run it" and reality -- much how there is a gap between a pilot being able to fly a plane and a pilot knowing how to maintain an Airbus, the airport and the runways.

It says something about the job market that losing a customer pretty much never happens. It’s so much easier to find sysops/devs than it is to find people who specialize in tooling.

Jira+Confluence, circle CI as the CI/CD service Artifactory for binary hosting and GitHub+gitlfs

circle had the lowest friction compared to Jenkins and GitHub actions wasn't out at the time!

I use to say that it’s like one of the business domain development teams, it’s just that the domain is “dev”. The businesses devs are the stakeholders, and the tooling/platform team members are usually already domain experts that write code.

Having a “devops” team would feel like having a team called “agile team” or something - you do “devops”, you can’t build it.

Too late. The industry has chosen to deviate the word.

For years as a sysadmin I was arguing people that devops is a methodology and not a function. Now I have the Sr Devops Engineer title. I know I am still just a sys and platform engineer who's job is to facilitate life of developers but hey the industry insist on calling me devops, I have other fights more important than fixing my title and team name.

I just tend to think that 20y ago people were relaying memes about sysadmins being motherfuckers who just randomly denied everything to clueless developers while the devops term emphases more the collaboration side. I hope the devs I work with are happy working with me, I sure have a good relationship with them. If that is being what the industry now choose to call a devops, I am a devops.

Realistically though, I'm 100% sure that absolutely nothing will stand in the way of the DevOps job title being denigrated.

Companies have for a very long time used title inflation to bait and switch the unwary software engineer into editing enterprise software configs, and will continue to do so for all of time..

And I definitely agree with the article here that if you're using PaaS, you don't need a DevOps team. And there's certainly a better feedback loop there.

But many will argue that PaaS _still_ isn't there yet for highly complex applications. Unless you go to a point where you basically need a team managing the eventual complexity that PaaS adds back to things for you to achieve complexity.

In any case, this is mostly pedantry. We should all strive for simplicity as much as possible, but complexity in apps breeds complexity in organizational structures. DevOps was never a silver bullet, and there probably won't ever be one.

Part of the reason that the name DevOps was chosen almost 10 years ago when it was started was to help signal to the wider org the cultural shift we were undertaking, and that the expectation was going to be for devs to own their systems all the way through production.

At this point, we would probably change the name, but the team is quite attached to the name after 10 years. We actually did try to rename the team, but everyone still called it the devops team and we gave up on the change.

A sysadmin would write the tooling to ensure the payment code could work (failover databases, redundancies) and put safeguards in place so that it does not get traffic that is not intended for it.

Sysadmins used to write the code that touched thousands of machines at once, developers used to write the more complex code which were the bread and butter of the company.

But where I’ve worked, it was never a silo. Just a different mindset, and everyone worked together to do the best they can. That was before “DevOps”.

Uhh if by tooling you mean a bash script to deploy ssh keys to prevent team A from accessing a machine owned by team B.

You really have no idea what you're saying.

A junior sysadmin executes checklists, writes simple scripts, reads logs, and solves common problems, all while learning the complex supersystem of their workplace. They learn configuration of systems in use and how infrastructure is arranged and commonly changed.

A sysadmin helps to create checklists, writes more complex code to work in an automation system, and generally solves problems with non-obvious causes and/or remedies. They know all the routine events in their workplace, and add to documentation. They build, configure, test and recommend new tools and solutions to users, devs and the rest of the ops staff.

A senior sysadmin works on policy, debugs automation, solves complex interactions, and has an excellent understanding of complex systems and networks, not just at their workplace but in interactions and hypothetical interactions with the Internet at large.

DevOps guys are senior sys admin in the cloud (and that last caveat isn't mandatory). People shit on sys admins all the time but the only real difference between senior sys admins and DevOps engineers is pay packet, tools and responsibility.

You have a dedicated team building an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Google App Engine (or Heroku, or Digital Ocean App Platform, etc).

I know this might be unreasonable, but I always get frustrated when everyone suggests that anyone running their own infrastructure is clearly doing it wrong. I get that most dev groups are creating web apps and should use a PaaS, but there are also a lot of us who are CREATING those platforms that other companies use. We also have to write and deploy software, and sometimes I wish I could have discussions about how to do that without most responses being that we shouldn’t do that.

Maybe general developer forums like HN aren’t the best place to have those conversations, but I haven’t found great active forums dedicated to platform developers.

There's tons of interest, but much of the knowledge is covered behind corporate NDAs/employe and getting people aligned on terminology and technologies is hard because so much is hidden behind NDAs. The current solution is either to work for a big company where there's rigorous internal discussion or go to tech conference after-parties and hope to meet the right people at the right time/place on the Ballmer curve to have a productive discussion.

Heroku's simplicity for deploying is honestly a thing to aim for, but using Heroku at scale while keeping things consistent and secure is extremely difficult. It's not built for large, complex infrastructures. It's built for small shops and ad-hoc deployments, and it's dope for that.

Which is one of the things the whole institution was designed to solve.

I realize that there’s quite a lot of devs these days that got in it solely for the money, and don’t care one whit about what happens after they throw the app over the fence.

But the whole reason I liked the movement was that it gave me more control over my own destiny. With half baked devops systems and processes that I have to follow it’s (mostly) no different from the BoFH of yesteryear.

It's sort of like expecting you can get by with only sales engineers. Having someone good at either job is hard enough, expecting you can staff only those that are just as good at both? Reality is you'll just end up with people deficient in one area or the other.

You might get by with a small core of people that can successfully straddle both roles, as long as you don't expect them to fully handle the needs of both departments, but I suspect hiring only for either department and fostering good communication and encouraging them to work together on projects that help deployment and availability yields better results. Maybe that partnership will grow you some good devops people if you're lucky. Maybe you should resist making a department out of them if that happens.

This is in an enterprise sized organisation with 10.000 employees.

We still have a sys ops team to handle security, network and all those other things, but deploying software? That’s really easy for our developers.

We use Azure DevOps, we deploy to Azure apps with a dev, testing, staging (5% traffic) and prod (95% traffic) slots. It takes maybe 5 minutes to go through the to-do-list when a new application or function or whatever is created and we have a very fascist naming convention. This isn’t an azure commercial, we’re only in azure because we’re the public sector and already so in bed with Microsoft that it’s the cheapest solution. I’m sure you could do the same in any other cloud.

Good service is ones that solve the problem in a reasonable time. Great services accomplish the same as good service but understand that the code, company and even the pipeline will evolve and either the client will need to reinvest themselves or externally again. They will communicate this with the limitations to the client.

There are only two types of clients I've seen more inline with the original statement, declining or flat company and both are only in the maintenance do not touch phase.

Nothing they build us would last forever, but then, we operate 300 IT systems, only a few major systems have a life cycle of more than 5 years. 5 years ago everything we ran was on prem but on VMs on rented hardware, 10 years ago it was physically on prem some on physical servers, now it’s in azure, who knows where it’ll be in another 5 years.

I fail to see what a full time staffed devops department would give us except for added continuous cost that is so much higher than its alternatives that it’s just ridiculous.

I mean, I understand full well that system operators are worried in the current climate, but most of ours simply developed into broader roles that aren’t becoming obsolete.

In my case, the team « responsible for creating and maintaining the tools and processes that allow the dev teams to manage their systems in production », was just a regular OPS team. So they built tools that were not working properly and they struggled to focus on the most valuable parts because they were not using them. At least, they were the one getting paged when systems were down.

My next ask for my project with my next client, will be to embed a devops full-time within the dev team. At least they will understand and share the pain of having to work with a git+terraform ticket system and all the bad ideas that our OPS friends think are great.

I just don't want them to stay in their ivory tower and keep breaking my balls with their broken terraform templates.

This keeps being trotted out, without question, as the reason for the shared devops platform team. So what if each team has their own opinionated build, scan, deploy tooling?

It's used for compliance security scans now, so we are never going back.

Think of home improvement. If my toilet is clogged, it doesn’t make sense for most people to call a plumber. That doesn’t mean that plumbers are bad, just that it’s easy for a generalist to do basic tasks.

Likewise “pure” devops rarely makes sense because developers are assigned to projects that end, and sustainment doesn’t require the same skills. It’s cheaper to pay for 20% of a $75/hr ops engineer than 100% of a $150/hr senior dev.

Some Developers tend to stick with the Dev parts, ignore the Docker/helm configs so, yeah there is some trouble with a true "DevOps" project.

And Google is re-introducing a "K8s stability role", which is a Sys(Dev)Ops role, pushing on that direction (for valid reason).

A true shift so it is difficult, but DevOps time-to-market deploy speed is so high I will no look back :)

That doesn't make your team a devops team. The devs are in fact the devops team. Engineer is ostensibly a better title so, because reasons, the engineers are just calling themselves engineers.

What you're doing is really internal tooling, dev tooling, infra, prod, those kind of team names.

Then there is the infrastructure/deployment team and finally the developers.

At least in larger organizations i think that is a common way of doing things.

The article suggests using a higher level abstraction with better tools suited for the developers than the low level tools as docker, Kubernetes etc. You can make this into a separate internal product and still keep the devops idea. But as soon you break out any of the core responsibilities for the product from the team. If that's business, security, stability or quality you are moving away from devops. Sure devops isn't the solution to all problems. But why call it devops when one doesn't like the basic devops idea.

IMHO I would guess they're too close to the dev people to care about the bigger picture

FYI, that is sysops, in case that you missed it