Somewhere along the way "sysadmin" became a euphemism for helpdesk, and some folks thought "devops" (as a job title) was new, but it was just the same old sysadmins really, or new devs becoming new sysadmins with the fresher title, they felt themselves the vanguard of a new paradigm, I'm sure.
Now we're in the same situation again, where "devops" (as a job title) can have no development experience. So we need a new term, apparently.
Maybe gatekeeping is useful to stop this job title treadmill.
This was a totally misguided re-branding of the field. It attracted bureaucratic personalities and classist managerial types, and utterly destroyed the field.
Like you, I couch my job description: I'm a "Systems Software Developer, I develop software systems" .. because its true, thats what I do, even if its a majority of the time working on embedded projects.
And if someone then responds with "ooh, so you're in IT", I immediately go to another corner of the party and avoid them like the intolerable bores they are ..
No need to be arrogant towards people outside the field. My grandma doesn't know the difference between the random IT guy that updates your Windows printer driver and a Linux kernel dev, but I'd still not call her an "intolerable bore".
Hard to convince management that a dedicated onsite IT would be required - especially when they think it is your job already.
Gotta appreciate if you're working in the former. Drawback is you can't BS your way through issues. Everybody above me in the chain (which is like 4 people) could write a quicksort blindly. If anybody tries any BS they don't have their job much longer.
Well, nobody forces you to work for a company with managers like that. I personally wouldn't.
That's kind of like saying, even slaves have a choice. Death being the option available to all.
Drawing a comparison to slavery is quite shocking, tbh.
It isn’t a problem for my ego to do that, and it’s something that’s both relatable and typically stops the conversation from revealing anymore about my work (not that there’s really anything to hide, I just enjoy my privacy and no, I don’t want to meet your “computer genius” nephew or fix your phone).
If I’m in a clearly technical group, I’ll go into more detail. I don’t identify myself with the DevOps term though, or sysadmin, or developer. I don’t clearly fit into any of the -currently assumed- definitions of those.
In social situations, I usually just say "I'm into computers". Most people who ask what I do don't really care, they're just engaging in ritualistic small talk anyway. If someone does care, they'll ask clarifying questions.
I always default to "I work in IT". If that person is in the space I might go into specifics; but even then majority can't relate or understand the role i spend my days working in.
It also has the benefit of coming across as more blue collar and I catch less grief, as the disdaine for tech workers continues to grow.
A good example is how software releases work. Where I worked 15 years ago, releases were something we could afford to do about four times a year, at night, on the weekends, doing the rollout by logging onto machines and starting them back up, babysitting the rollout. Because rollbacks were catastrophic, an insane amount of time went into QA, so the time between a code change and a release could be easily multiple months. Now I'm at a place where releases are done every day of the week. Rollout steps are largely automated. Rollbacks are mostly drama free. Risk is mitigated through automated canarying (and automated testing before that).
And certainly the business has successfully gotten rid of paying people to log in to servers on a Saturday night to perform a release (which frankly not a lot of people are into in the first place, so everyone wins).
https://www.linkedin.com/jobs/junior-devops-engineer-jobs/?c...
No programming experience required (only some minor scripting in shell languages & a reference to education), in fact this job ad reads identically to a sysadmin job from 2014; in the middle of the "sysadmins can't code" zeitgeist, when all developing sysadmins had moved to become devops engineers or whatever.
Copied from the ad:
----8<---- Your tasks:
* Containerization and orchestration of our products for development, testing and production environments designed for cloud and on-premise solutions
* Produce reference architectures and standardize solutions to streamline efficiencies within the business
* Mitigate against Cybersecurity threats by ensuring systems are safe, secure and compliant to best practice recommendations (e.g. CIS Benchmarks, Vulnerability scans and Penetration tests)
Your profile:
* Finished studies in Computer Science or software engineering (Bachelor, Master)
* Proficient understanding of virtualization, containerization and orchestration (e.g. Kubernetes, Podman)
* Experience working with CI/CD tools such as Azure DevOps, Jenkins
* Experience of Infrastructure as Code tools such as Ansible, Terraform, Helm
* Scripting knowledge including bash, PowerShell
* English fluently, German is an advantage;
* Improvement-driven, structured and proactive team player, high sense for quality, open minded
* Working in an agile environment (Scrum, Safe)
---->8----
Proficient understanding of virtualization, containerization and orchestration (e.g. Kubernetes, Podman) Experience working with CI/CD tools such as Azure DevOps, Jenkins
The point is coding skills are not required to do the job. Sure it would be nice but if somebody ticks all the other boxes then being good at programming isn't important.
What I'm trying to drive home here is that you do need programming experience to be a good sysadmin/devops/platform; back when devops as a job title was becoming "a thing"(tm) I was told over and over again that sysadmins couldn't code and thus we need devops.
Now it seems devops don't need to be able to code (which you seem to think as well). So we will come up with a new term to cover programmers who understand systems.
Those "degree" sections are universally optional, they're always listed by I have never seen them honoured.
I'd put $20,000 down on the wager that I could get that job with only a Cisco certification (or even nothing at all).
But the criticism was that the job ad was not requiring knowledge of how to write code, and I was pointing out that the degree requirement (strict or not) indicates that that's not true.
If you don't have the required degree but can demonstrate equivalent skills, then that's fine. Kudos to the company about being flexible on the formalities side.
There is a line that is universally ignored in job ads. This is that line.
if I showed no ability to write a for loop or a linkedlist. I could still get that job.
I see it time and time again, it's like we must add that line to job descriptions, but it's never checked, never verified and ultimately it serves no purpose. There will be no difference between someone with a BSc in CompSci, an MSc in CompSci or no degree at all.
The thing that will get you that job, likely, is discussing how http works, how tcp works and perhaps something to do with exit codes.
Might not be true for this particular posting.
The only european country that takes titles even more serious is Austria. So without a M.Sc. in CS or a similar M.Sc. your chances may be quite low to even get an interview. Especially since university is basically free here so many people have a B.Sc. or M.Sc. in germany.
No BSc or MSc here.
I’m sure this is the case in some places. This can’t be farther from the truth in many others. At every organization I’ve been a part of, the best “DevOps” on the team was usually the person who was the best at coding, or at least understood it the best - and coding is one of those things that you can’t really understand well if you don’t do a lot of it.
But plenty of developers only have a vague picture of what production looks like especially in a complex environment. However they are perfectly able to do their job okay and produce working code.
Sure, maybe some of the work isn't writing software. But some of it is, and it shouldn't be written by amateurs. Also, they are building these things for people who do write software. It would be hard to produce something that works for them without understanding what they do.
Reading code, understanding building and deployment are all useful and required to some extent (especially the latter). But specific programming skills are not required any more than that would be for say a technical writer or network engineer.
That typically entails programming experience, especially in Germany.
Most companies do. Perhaps not top Silicon Valley tech companies with cutting edge needs and paying over $150k, but overall I've found that dev ops teams have plenty of jobs for 'green' team members. You just have to accept a 'green' salary and 'green' job responsibilities.
Yes. Devops people come from a SWE background generally + they understand infra. They usually have 10+ years experience. Most specialties work this way in software: security, devops, AI, ML, ... etc. You need to be a SWE first, then specialize.
Also I work in AI/ML and hardly anybody has 10+ years of SWE experience before specialising. Most people start fresh out of a PhD or postdoc program or come from some sort of math or physics background.
Now it's actually 80:20 due to software engineers becoming frustrated and disillusioned with the current state of things. Not only due to specialization and the push towards tool standardization (you don't get to build many custom components because we k8s all the things so most of the time you're just writing yaml/hcl), but because working with regular sysadmins gets extremely frustrating.
This could be the entire problem. These are specialized fields. They are being treated like they are not.
[Weeps] That wasn’t the point, the point was exactly the opposite of that, to get developers and ops working as a single team, building an understanding of each other’s domain in order to make things better. Sadly the point was lost long ago, and devops engineers and devops teams have taken the place sysadmins and ops teams in providing a nice little silo you can chuck your terrible code over the wall at and not have to think about how it works.
But you know what? There are tons of developers who are _perfectly fine with this_! There are tons of developers who just want to write business code in their CRUD webapp and call it a day, the more abstractions you can give them, the happier they are. And while I don't like it, I don't blame them.
the ole “runs on local therefore it’s good to go” should be good enough.
Which database you choose and how you scale it will affect your application code.
You can't write code that makes use of the local file system (or in-memory data for anything meant to be persistent) if you have more than one container serving your app. (And yes, some people need to be explicitly taught not to do that.)
And so on.
Where this falls down is when development teams are building complex systems and then expecting the ops team to divine how that’s meant to work in production.
A lack of gatekeeping I don't think was ever the issue; the incentives were just perverse.
I spent time working around the space industry. Almost a decade of my life.
The big contractors in aerospace have a _really_ difficult time finding people who know enough about the hardware and the software to be competent engineers. They’re basically unicorns.
Not that they don’t exist, but if you’re looking for someone who is demonstrably good and has a validated educational background in both, you don’t have a huge pool of talent to go with.
So they hire them when they find them, but they also just tend to hire experts in each field and let them collaborate. They didn’t try to reinvent the wheel with some new “paradigm shift”.
What tends to happen is that at some point in time, an organization finds itself in a bad place. Where pagers fire regularly, and the skills you need to manage the noise are "fast context switching", "good communication", "debugging", "ability to work insane hours". Very little of this involves coding, or building software. Teams may start by assigning some engineers into this role as an elite team, but if you come back in a couple years you'll find that the elite team has burned out or forgotten how to write software and due to the impossibility of hiring new elite team members - has resorted to hiring people to carry a pager.
There are a few companies which invest in big platform orgs, but due to the lack of revenue generation - it seems that this can only be sustained with very strong leadership buy-in.
I absolutely agree with this. It seems like a it inevitable for anything that a business needs to do but doesn’t really want to do. Another place where you see it is with project management, where there have been many different titles for the same role.[0]
Definitely. often it's more about organizational changes than software and upper management is the one blessing things in most big corp.
One of the fundamental shifts in my opinion was building APIs on top of sysadmin tasks. Provisioning a VM, installing an OS, getting software onto the server, configuring ingress/egress, handling process restarts, etc.
These all were turned into tasks you could run from a developer’s laptop.
But it was still a different skillset and devops dumped that skillset on developers (“Hey! Now, instead of opening a ticket, just write a 1000 line magical incantation of poorly understood YAML and the system can do it for you!”).
Platform Engineering grew out of that pain. It’s the layer we are building on top of devops in a way. We are taking those APIs and treating them as foundations to build products that manage the SDLC lifecycle for developers.
I think at its core, deveops was about “how much sysadmin work can we turn into an API and give to developers?” The answer turned out to be almost all of it. Then Platform Engineering is “now that we have APIs, how much can we automate away for developers.” We don’t know the answer yet, but suspect we will find the answer is also “almost all of it.”
Site Reliability Engineering seems to be following a similar story arc and converging on Platform Engineering.
At my current company theres literally no coherence. Some people deploy on EKS, or ECS, or Anthos, or regular linux hosts. We use every database on earth. Things on Azure GCS and AWS. Databricks and Snowflake. The list goes on!
The team I was on was made up of ops/engineering[1] people who knew how to code. We built tools to fill gaps or let us do things we couldn't do otherwise, but would typically use out-of-the-box functionality in third-party products by default because it was cheaper in the long term than building and supporting custom code. Maybe there's a better term for this, but I think of it as "OpsDev", and AFAIK it's more or less extinct.
DevOps has always struck me as being developers[2] getting fed up with having to deal with a separate ops team, and building tools and platforms to automate away that ops team entirely.
The two things can look superficially similar, because they achieve similar goals (automate away repetitive work, let people do more work), but the approaches are almost totally different. Kubernetes and Terraform do amazing things, but they are very obviously tools written by and for developers. I think it would be difficult and unwise to hand off first-tier support for issues involving them to a help desk, for example. It's pretty safe to write instructions for first-tier support to use a GUI or run a command but substitute the name of a server farm and a software package in the arguments, especially if the command is a script written by the engineering team to verify that the work makes sense before executing it. It's a lot less safe to try to have non-developers hand-edit source code, a script or even a YAML file and push the result to production.
Both approaches work. They just require different kinds of people, and each has their own benefits and drawbacks, and IMO it's a mistake to put them both under one label.
[1] "Engineers as in Geordi LaForge or Montgomery Scott": the people whose primary job is to keep the engine(s) running, respond to emergencies, make improvements where possible, and generally be a smart-person-of-all-trades.
[2] "Engineers as in Burt Rutan": the people whose primary job is to design and build new products (for lack of a better word) from the ground up.
First of all platform engineering is two words. second of all what is my understanding that devops = platform engineering? Do you have a dictionary definition for these? No, so it does not make sense to say that you should use X instead of Y. Btw. systems engineering and technical operations are the things we usually mean by devops, deploying, configuring and scaling applications in an IT infrastructure and dealing with different aspects of running these applications. DevOps was just an approach to remove the silos that some companies built between development and operations bringing good old tribalism to technology because we cannot exist without tribalism.
> Let’s get rid of devops and hire platform engineers.
I wish this problem was so simple.
> DevOps was just an approach to remove the silos that some companies built between development and operations
I know this is always the intention but it is never implemented this way, it becomes a “throw crap over the wall” situation again because now you’ve made a second team and management sees that stuff as “their” job and the silos develop.
I don’t profess to know the correct way to do it, but if I was managing an IT team, I’d assign a dedicated “Ops” guy to every dev team who would occasionally also write code. I dont care what you call this person. I’m in favor of platform engineer because it’s a lot more defined than “DevOps.” People have wildly different ideas about what “senior” devops is too. With a term like platform engineer, my gauge for if a person is a senior at that role is by the quality of the platform they can deliver to me. it’s vastly better, but again, I don’t give a crap what you call it as long as organizations finally understand what they’re hiring someone to do.
I like the term platform engineer or maybe just infrastructure engineer, but I don't think they really add much specificity. For instance, we don't use any of the big three cloud providers, which I suspect would be an expectation for most of these titles. I think the possible diversity of modern tech stacks is just too much to pin these titles down, and mostly I just tell people I'm an "IT guy" ; it's only when you talk to someone in the industry that titles like Devops or Platform Engineer have any meaning, and as this thread shows, there's not even that much consensus within.
DevOps is similar to agile. Originally they both described a set of principles, an ethos, not a specific role. DevOps was your development team work closely with ops as one team, using software engineering practices, config as code etc etc.
Then, it just came another name for sysadmin and later platform teams and back to where it all started, passing things over the wall from engineering to platforms.
The way I understood DevOps (back from the first DevOps Days in Ghent) is that clear separation of sysadmin and developer roles harms the process of software delivery and that developers should maintain their code up to production, and learn from the process.
It was, in a way, a critique of the project-driven business where teams ship and jump over the board.
Then it turned into this guy who can not really do anything. Can not really check in code because the devs do that (separation of duties). So you never get any real experience in the codebase. Can not make any decisions because the marketing/product owners do that. Can not really make better test cases as that is for QA. You can however help on the helpdesk they are short 3 people today. Then the meetings. Endless meetings upon meetings. Because your 'devops' you need to know everything that is going on and better have those statuses on the high priority items to be fixed. Oh and you are 'on call' so you can never really 'go home'. Then if something breaks all you can do is just call in others and make them fix it. Making you little more than tier 3 helpdesk. Useless paper pusher job with zero authority.
I enjoy making sure software works for people, the more people the better. If that involves all that sysadminning, I will enjoy that as well.
But most of my peers are oblivious to the fact whether their code will be used at all and are having fun filling their heads with complex abstractions right before spitting them on screen.
Point is, once developers start doing support, they stopped taking those learnings and going back and developing better. Instead the organization just started having developers do support. But then, why are the support people these highly paid developers, lets change the role definition again to not need to code.
Seems like on-going title shuffle, around, we don't really want to pay to develop, why can't we get by with no-code.
Here’s where it went wrong.
> “devops” has literally no meaning.
No, it does, you’ve just chosen to let vendors, consultants and managers redefine it as “whatever we were already doing”.
I'd recommend the lead takes the role.
Align incentives to create simple, reliable software.
This sounds like an incredibly toxic work environment
This is the way.
If you want to cut through all the BS, this is ultimately a debate about cloud vs on-prem relative to our ability to manage complexity across multiple vendors & styles of product. If you go to the cloud and actually do it entirely their way without antagonizing the process with 3rd parties and other ridiculous nonsense, you don't need leet hackers to figure things out for you and build elaborate processes that you'd require a "devops" team to support.
In all 3 major clouds, there are well-documented happy paths that a junior engineer can follow to put a perfectly functional webapp live on a TLS-secured domain in about 1 business day. As far as I am aware, "getting the site live on public internet and making sure the certs are ok" is DevOps bread & butter. I am not going to pay for a dedicated team to do this kind of babysitting in 2023.
The real tragedy is all of these companies who "went to the cloud" but just wound up in an even more complex & contorted hybrid stance with the cloud simply appended to their vendor list.
"By leveraging the capabilities of a multi-trillion dollar Death Star ..."
The objective is to get completely out of the game of managing VMs, certificates, authentication, MFA, et. al. There is no way in hell we can do a better job than any cloud provider when it comes to compliance, consistency, security, etc.
Maybe when we have 10k+ employees we can reconsider the benefits of suffering additional complexity.
That's a hell of an assumption.
> there are “easy” plug and play solutions offered but they cost a lot and come with tons of drawbacks.
Agreed. But, I'd argue that these drawbacks/constraints are exactly what you need when you are faced with a virtually unlimited # of choices. When you are building software that needs to survive audits and is sold to financial institutions with 5+ year contracts, intentionally paying to have your hands tied is a fucking blessing.
I also used to shit all over the cloud until I was put in charge of the whole product stack and had to field difficult phone calls from other CTOs. You may find your position change over time too if you are put into this kind of a situation.
The AWS Lambda + Serverless framework shit trend has to go. There are no savings or agility. It never handles everything. The gaps are the painful parts, not the bits it gets right.
I'll choose a simple nodejs app any day over that yaml trash fire.
One thing that was very different was provisioning new machines consisted of ssh’ing in and installing the required packages and binaries. The deployment was probably a simple script that used scp.
However, once the term DevOps was coined almost immediately some people began to stop doing development work and just focused on automating operations work and constructing glorious crystalline castles in the clouds to handle traffic that could have been realistically served by 1U in a shelf somewhere. Complexity begets complexity.
In our arrangement, the ability to push code to production is gated by the GitHub/Azure integration path. The QA or project person who is rotating the production deployment slots (azure functions) is not granted access in GitHub to deploy to those same functions.
So, the developers pushing code and those deploying code are mutually exclusive groups. You could still defeat this with collaboration between employees or screwing with AAD records, but that's why we have a ton of audit logging turned on too.
Absolutely, you need proper controls in the system to control and authorize deployments with appropriate responsibilities in place, but fundamentally if your deployments are fully hands-off, it shouldn't matter who clicks the 'deploy' button
I’ll add that platform engineering is where much of the hardcore engineering happens nowadays in modern software stacks. In the olden days, SWEs had to know the hardware/low level software stack themselves.
I think the SWE role is what is at risk of commodization. It’s something sufficiently good LLMs may even be able to do themselves at some point given the right instructions.
Platform engineering will never go away - it’s close to the hardware and so many things can go wrong. You’re building the abstractions so SWEs can sit in their cozy IDEs debating about overly complex language features and libraries (said tongue in cheek as I’m guilty of this plenty).
What is an "infra platform" ?
TIL about platform engineers. But I'm still unsure what kind of things they produce. I know that they basically create a self-service organisation-specific infrastructure that can be used by developers inside an organisation to deploy software. But how does this self-service look like? Do developers fill in a web form to automatically get a server with the right firewall rules?
The whole idea behind platform engineering is the mindset that the "platform", i.e. the stuff that runs the things that business cares about, is a product.
Reframing the platform as a product instead of "IT" or "the place where apps go" introduces three concepts that were previously foreign to operations:
1. The platform has customers instead of consumers, and getting/acting upon their feedback is important,
2. The platform has some semblance of reliability guarantees, and copious observability is needed to uphold them, and
3. Like other products, the platform needs a dedicated, cross-functional team to maintain and improve on it.
So basically platform engineers are engineers that focus on either the "developer experience" part of the platform (i.e. the platform's frontend) or the platforms back-end (servers, persistence, networking, etc.).
> But how does this self-service look like? Do developers fill in a web form to automatically get a server with the right firewall rules?
Kind-of. In an ideal platform, developers wouldn't even know or care that their stuff is on a server. It could be in a toaster, for all they care. So it's kind of just there. They write and maintain code locally, and their build and release pipelines stores and deploys into it.
I do not think my organization would call this a platform engineer, as that title makes no sense for the work I do.
damn, that sounds pretty good. need to look into some devops courses.
It always has been. Always!
This is why FAANGs paid their SREs top dollar.
The problem, as you noted, always starts from the top. Cultural change needs to be championed by someone with power inside of the company. However, cultural change is slow (site starts to scale) and expensive ($150k hacker nerd) and many executives still think of tech as a cost center to be optimized (see also: ChatGPT).
Platform Engineering now is where "DevOps" was in 2014 or so, but with more Kubernetes and Rust. But as long as executives who do not (or refuse to) understand the value that strong engineering teams bring forth, the Platform team will just be interchangeable with today's DevOps team.
Can you be more specific?
From what I understand, "DevOps" helps you get what you want into production. Whether that be a few Docker containers of your node.js or Rust or Go or Java or C# or whatever application (probably an HTTP microservice or a batch job) as well as your infrastructure (Redis, Postgres, a message queue maybe)
Obviously scaling your microservice API from 2 to 4 isn't "all devops do". I also get that not everything is a Docker container. I'm just curious what I am missing.
I get that for example, making Postgres "scalable" is much harder but... is that really a "devops" thing or is that a Postgres-specific thing?
Same for Redis if you need multiple scalable instances.
Scaling infrastructure isn't easy but I thought devops was just responsible basically for standing up the "pipelines" to deploy whatever you want.
Why can't you just have a monorepo of application YAML that goes into ArgoCD or something like that? Where do you hit the "need devops/consulting" part specifically?
I think they might not “need” it as much as they think they do, but they want faster deployments, better automation, less downtime. Usually the devops team is the one that handles this for most organizations. ramming code into prod by scp’ing some files into a single monolithic host can get you a long way, but at a certain point, it doesn’t, IMHO.
Which part of that isn't covered by `kubectl apply / argocd sync` in a pipeline of some sort like a Jenkins or a TeamCity or a GitHub Actions?
To me that is simply not DevOps. That's just Operations. Businesses are throwing around the title without a care for the word
In reality I spend my day doing everything from yes infra work and living in yaml, to working in Go/TS/ruby/elixir/etc repos and building those as well.
We have dedicated infra team, and as platform engineer I have to know their job to an intermediate level in addition to having advanced level in shipping code across stacks.
I love it but most other people seem to struggle with the constant context switching.
I am entering into my third major project in a row where it’s almost the exact same situation (and same consultant I’ve bumped into)
- site starts to scale, never had an ops team
- realizes oh shit we cant manage this ourselves but who wants to hire some hacker nerd for $150k a year that “gEnErAtEs nO BuSiNesS rEvEnUe” so they hire a consultant or consultancy firm
- firm builds them a solution that will technically “scale” but no one in the organization bothered to learn how it works
- it works for a while til it doesnt and then the “oh shit” moment becomes “oh shit we need a dedicated devops team”
- devops team attempts to work with the shitty consultant solution (editing yaml) and does an okay job until something breaks horribly again or they become a massive bottleneck
- finally they allow someone to come in and clean house and you get an infra platform like the author of this blog is describing
Platform engineering is the right word. “devops” has literally no meaning. I’m only 6 years or so into my career, but I can count on a single hand the amount of devops engineers I’ve worked with that can actually “dev” - and by dev, I mean do stuff like dig into application code and suggest modifications for the infrastructure, or write their own full fledged libraries.
Let’s get rid of devops and hire platform engineers. The problem is it’s a really weird and rare combination of skillsets, in a field no one really wants to do except complete weirdos (I am one). I’ve long said that DevOps isnt a career you choose, it just happens to you.
Well said! thank you so much for this post.