I'm a developer, so I'm very biased.
My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...
1. They could give you root access. But then you might start making other changes, and over time that makes everything unmaintainable. By them just typing in what you want this one time, they limit the scope of your access to just one change, they are aware of what changed, and can push back on it. It's basically a crappy change management process.
2. This person has a lot on their plate (a way larger backlog than the average dev) and doesn't have time to research how to do the thing you want them to do. They are asked to be experts at everything, but they can't be. So they ask you to show them exactly what to do. Gets rid of ambiguity, lets you both troubleshoot any unexpected problems in real time.
3. A lot of roles where one person/group has "power" over another, often takes it for granted and unnecessarily gate-keeps and adds red tape and blockers. DevOps is supposed to flip this situation on its head by giving you, the dev, the tools needed to do your job, without you making changes to the system that would affect system stability. This keeps the system stable while giving you agency. Unfortunately, doing all that is much more difficult than most people realize, so it rarely happens.
4. Like every role, sometimes there's idiots. I've met my fair share of both idiot devs and idiot sysadmins.
2. Yes, I'm sure this was the case.
3. Indeed!
4. I haven't! My original comment just took some situation out of context. I know nothing about what else the sysadmins were doing, they probably had other things on their plate about which I knew nothing...
When it comes to debugging running systems, nobody has it better than doctors. They knock the "system administrator" out and go right in with root access via scalpel to do what they know needs to be done.
SysAdmins can be a huge help in devops culture, but sometimes things go wrong and the surgeon needs to get in there.
But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?
> "Is it never necessary to run ad hoc commands on your production systems?"
Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team.
Even the most competent people, sysadmin or developer, can make a mistake.
I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.
SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host.
"Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it.
Ironically, these examples are only proving the point made in first comment.
I have however worked with a narcissistic power hungry incompetent sys admin in the past. That doesn’t mean though that I automatically assume that all sys admins are the same. The one I am working with right now is highly competent and a pleasure to work with. And he is smart enough to simply give VM’s with root access to developers who ask for it.
The technical stuff changes but Systems Administration remains the same practice. You figure out how to put together a complex system, how to maintain it, keep it highly available. Operations is the work of keeping the business afloat through IT, and Systems Administration will be a part of that as long as there are Systems to Administrate. The title will change, because apparently people are squeamish about being lumped into a group with people who aren't all cutting edge. But the art won't be lost. It lives in every woman who stays up til 3am to ensure maintenance goes as planned, and every man who tells the developers, "No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. ...."