"Yea the DevOps guy messed up the widget and nobody notic---"
"Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?"
"Bro that widget IS DevOps."
-silence-
Same applies to the topic of "micro-services".
I think that's where most people go wrong. They put a bunch of services in front of a shared database, which means that they don't have to go through a service's API to get to it's data, and that's what breaks everything.
If you see a queue between two services, that is usually an indication that some ownership is being transferred (even briefly), and that is a critical point where you need introspection and observation.
> An important rule for microservices architecture is that each microservice must own its domain data and logic . Just as a full application owns its logic and data, so must each microservice own its logic and data under an autonomous lifecycle, with independent deployment per microservice.
https://learn.microsoft.com/en-us/dotnet/architecture/micros...
I don't recommend either, and it's still a code smell, but with clear definitions they can work.
Btw some of the confusion might be what I mean by shared database. I didn't mean two services sharing a limited set of tables, using the database as a rabbitmq replacement. I meant sharing the backing database of the microservices. It sounds like we probably agree I just wasn't very clear about what I meant by sharing a database.
(Ironically I am literally using a database as a queue to share data between two services we are running in prod. I don't think of it as a microservice because it's not separate teams, it just because our monolith is hosted on a platform that doesn't support certain libraries so we had to role those libraries onto a separate platform that does support them.)
Using a shared database allows you to punt a lot of that complexity to a system that's been specifically designed for it, and working well for probably 20+ years.
Too many people think microservices don't have their own, severe, downsides. The likes of netflix, google, et al, can afford to pay people whose entire job is to manage the complexities of these approaches that flat don't exist in other scenarios.
But it's a hell of a lot simpler to use a single database if you can get away with it.
Amazon's SOA architecture at least all started out about pulling bits of data spread over hundreds of servers into services that could take advantage of caching.
http://resources.1060research.com/docs/IntroductionToResourc...
Ok, I've seen this, but IME (24y in industry, the last 6 as a consultant) in the vast majority of cases, it's more like "things devs should control but CAN'T [bc CICD etc are silo'd and owned jealously by an overburdened ops team unable or unwilling to facilitate self-service]".
It only takes a few instances of people royally messing stuff up in production for their deployment rights to be stripped away – and other groups catch the fallout too, as new "procedures" get implemented.
Sure, there are some overbearing procedure monkeys who really want a process on everything, but a lot of "protection" rules are there because something really bad (and expensive, financially or reputationally) has happened before.
Having something not work during the development cycle is annoying for one person or a team. Having something break in production usually means people get paged and (depending on what you do) the resulting issues can have major societal aswell as economic impact.
In the ops world, this is especially visible if you go further down the stack. (from applications down the the network).
Network architecture moves at a toad's pace compared to webdevelopment, which is a good thing considering breaking the network will usually break every other system inside a IT department/landscape.
There was an angry Ask HN post a while back about how terrible the "new guys" are that are out there.
It then complained that the new hire wrote a horrible authentication service.
I thought it was an intentionally absurd post about the expectations put on new some rando new guy to write something important / they shouldn't be working alone on ... but they were serious.
I never miss an opportunity to share my favorite piece of comedy this decade:
There is a good talk that is also comedic as it comes from the perspective of someone who wants to fail at doing microservices.
Some teams historically worked fine together, others didn't.
I think the tools we have these days are much better (for ops) to ship what works on developers machines.
But if your organisation had a culture of not working together then devops didn't really do much.
and anyway, you've leaned into my point about it meaning different things to different people.
Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969!!!" :-D
Agreed also at rewriting of history when it comes to development/support/operations models. My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)
As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all.
My personal favourite is how many large multi-nationals are now building in-house clouds?
WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?
A couple million in salaries and bonuses.
An in-house cloud sounds like the mainframe installed in the raised floor computer room at the school district office I worked at in the late 90s; of course, the 12 foot long Unisys mainframe was replaced with a Unisys 4U pentium pro box pretending to be a mainframe, and then there was a lot of extra floor space.
If you're running your own 'cloud' in a (shared) colo, I dunno that that's really in-house. I guess it's still 'private cloud' though.
Can't call a mainframe a bunch of buzzwords like "Hyper-converged, high availability, on-premise software as a service cloud platform".
https://www.ibm.com/ca-en/products/z16
warning: buzz word alert!
“There aren’t any new problems. Just new engineers”
- A sig I read a long time agoI still have the exact same problem with respect to never having exactly the ratio of CPU to memory that would make my app happy.
The interface to the shared-use datacenter, if you're lucky, is a spreadsheet that declares the static resources you own and a remote hands guy that can tackle things beyond the capabilities of your remote KVM. If you need more capacity you need to work with the datacenter folks to order physical machines that might show up in a few months.
The interface to the in-house cloud is an API. In most instances, developers are completely abstracted away from the physical infrastructure and don't need to take a lock on some human in the datacenter to get their work done.
Not to say your scenario isn't valid and real, but I live that scenario every day today with in house cloud and virtualization too - it takes months of approvals and solutioning and security assessment and network engineering and procurement and costing and whatnot... To deploy a windows vm.
Because it’s missing all the glue. I need to self-service the VMs, the database, the load balancer, the dns records, certs, and hook them all up so I can receive production traffic all via an API I could theoretically do in Terraform.
I am guessing that in the 80s you weren't writing Infrastructure as Code to define exactly what resources you needed for your software, having it all set up automatically, and so on.
https://en.wikipedia.org/wiki/Job_Control_Language
True, they didn't call it "Infrastructure as Code" but it was used to define mainframe resources for jobs to use when they ran?
I mean … sometimes I shake my head in wonder, and other times I just shake my head.
1. Push code to an automated pipeline
2. I understand that the automated pipeline may need input from me in order to run successfully.
Beyond that, you need other people building the "platform" that they're deploying to (for the most part). Probably the ideal example here is Heroku.
Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.
1.) What's probably the traditional view which is breaking down the walls between devs and ops, developers carrying pagers, etc. I.e. at least in an idealized world, there are no devs and ops--only DevOps.
2.) As you suggest (and which probably more closely matches how "DevOps" works especially in larger organizations), an internal (or external) operations team provides a platform that developers can use. Developers are still going to be exposed to some operational details, but a lot of them are abstracted away.
In my opinion, this last subset just doesn't work well for a lot of branches of operations.
Sure, it might work if you run stuff in AWS and have an operations team managing that, But what if your team is responsible for things like Storage Arrays, networking equipment?
Doing continious integration and code review on some of these is hard if not impossible
This becomes a lot more apparent once you're dealing with B2B customers and their nonfunctional requirements.
We've had situations in-house when devs built some simple microservice to handle some connection to a customers BI/DWH system - it's just 2 days of spring boot chugging to wrangle APIs around without state, nothing bad. But then, the customer started blasting that team with SLAs, backup questions like RTO, RPO, retention, regulation adherence of retention, the whole stack of IT security down to the physical access control.. That poor PO was caught just like a deer in the headlights.
This is some operational responsibility we're taking over for our dev-teams. We're providing the persistence, and this includes a defined backup and recovery strategy, a security strategy and such. And this also includes experience in dealing with these insane questionaires, and correctly pricing absurd requirements for custom backup solutions. And in fact, "we will have to schedule a discussion with ops about this" has ended quite a few of these requirements with "oh.. it's not that important". Intimidation with long job titles and appeals to external authority do work.
This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.
This is a great way to do it. There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. Being direct about roles goes a long way towards simplifying corporate complexity.
That's Conway's law.
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."
I'm sure we're due for a new one soon which will follow the same path, since people seem to be admitting that Devops has problems on here more and more. Will that cause deeper reflection? Probably not for many orgs, because soon some consultants will dream up a new brand of silver bullet to slay the immortal monster of organizational dysfunction and that's much more exciting.
My favorite example is that of a restaurant. You hire waitors, diswashers, bartenders, line cooks, prep cooks, hosts and hostesses, and managers. And your waiters might be 100% focused on taking orders during prime dining hours, but at the end of the night, they help clean up assuming other roles. Your bartender might be busy making drinks, but they take a table or two if they can. Your prep cook might finish the prep early, and come on the line to help out. Your manager focuses on expediting everything and keeping the ship running smooth, but also can fill holes at time.
You wouldn’t expect a bartender to table every person, prepare all their drinks, cook all their food and clean up their table.
But that’s kind of what we keep trying to do, because we have new bartending tools that make it so that you need fewer and because we claim communication is harder than doing everything in one role.
Then when that person invariably gets overloaded; we hire more bartenders and complain they’re not good enough at cooking your specific dish.
1000% this.
I think it's an instance of how the conversation in software always talks about solutions as if they're end-all-be-all answers, rather than explain a problem in specifics and why that made the solution the right answer at the time.
Then that "solution" becomes a buzzword version of itself, and popular buzzwords are tools leaders use to overcome institutional inertia. Which is a good thing, to get over that inertia. But then what comes misses the insight and understanding that eventually turned into a buzzword.
I always thought DevOps the “function” just meant this, and being a DevOps engineer at a small company meant you did these with decreasing emphasis, where by the time you’re in the backend it’s just helping enforce logging, tracing, other observable components.
Has worked for me in hiring and being hired and almost everyone I know understands this.
Then they also tend to have to do their own security and spec out their own tickets and then do Project Management.
Most tech people should be called DevSecOpsBizAnalystProjectManager.
Two intelligent, sincere, experienced tech people can discuss devops, in detail, for a good length of time, and still be talking right past each other without really noticing.
Then came DevSecOps. Around that time, I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.
This reminds me of a joke my networking collegues used to tell to the new hires.
Welcome to the networking team, where we are responsible for the network, aswell as anything that runs on the network because developers never bothered to pay attention in their networking class.
It's like they'd never heard a BOFH tell someone, "Go away now, or I will replace you with a small and uncomplicated shell script."
After a lot of discussion, we mostly realized that we get the most value if we define more specific operational roles. We now have the idea of infra-ops, and product-ops. infra-ops is providing a deployment platform - a container runtime and persistences. product-ops on the other hand is responsible for deploying and running the different products from development on this deployment platform.
And this is giving people good ideas. Some products are very simple without harsh requirements. In these cases, one of the backend devs just takes over the product-ops role by setting up a deployment pipeline, a job and some migration handling. Other products are bigger - for example we are providing some of our systems as essentially managed systems to invididual tenants and customers. In such a case, parts of the product operations is with the product team - such as writing jobs, releasing containers and artifacts, and other parts of the product-ops role takes place in the managed services consultancy. These products are currently looking at wordings to split up the product-ops role according to their needs in their context. And that's totally fine - the infra-ops role is also divided depending on the system the engineer is working on. A postgres admin is an infra-operator, for example.
And all of this results in a devops oriented culture of cooperating across team boundaries through automated processes.
It's ok to have build-engineer doing release engineering if it's simple enough, but doing build engineering properly (Source control, artifact control, shared caches, dependency management of the compiler outside of the standard toolchain) is quite a large job.
Release engineering itself gets complicated when you take into consideration the different target platforms, most games get published on Playstation, Xbox and one of a handful of PC platforms (notably steam), plus the online systems and of course the development environments and internal release systems which are ubiquitous.
Build Engineering usually is the step between developers and QA, and release engineering is the bit after QA.
Both interact a lot with QA so could be rolled into the same role.
Fewer titles can avoid these discussions.
If they're not doing it or was overlooked for some reason, then that's an issue but not really an argument to not do it.
That won't be fixed with a new job title.
This is always a dead giveaway that something is a buzzword
Same for rest. Sometimes when people use it, it just means JSON + http requests. Other times it's supposed to be some kind of architectural style
Especially seniors and beyond. They force leetcode interviews they somehow pass or are grandfathered through and gatekeep "trash devs" by slamming gotchas about how the whiteboarded code discussion doesn't technically compile and how "returnAverage()" isn't a method ("did you know you have to write methods before they work? What does returnAverage() supposedly even do? Return a random character in the alphabet?")
I get excited about working in different tech areas. I'm exceptional at fixing other's bugs and maintaining code. Absolutely nobody seems to be hiring for this though. It's all about college exam trivia and leetcode.
When I do get into these companies, I have to work with people throwing around casual racism with HR joining in and f-slurs like it's corporate 4chan. I ask if anyone can help look at a critical bug I discovered, and nobody speaks up. Even when it turns out it was their last code change that caused it and they are the sole master expert in this area, and they were just working on organizing their desktop instead.
"I can try to look at the SQL problem, since nobody else spoke up." Then I'm explaining the basics of SQL to some guy sitting around blank faced and it turns out he was hired because of his 10+ years of expertise in SQL. His whole job is to tackle problems like that.
Meanwhile I'm out here looking for jobs when panic cuts happen and it takes forever because I'm drained from all the gotchas and gatekeeping in these interviews.
There's no way I could live with myself if I put this experience onto others. I lose sleep when I fail to call out someone talking over a quiet person on my team. I've never seen anybody else stand up for anybody, however.
I just kept asking questions until I found the one she missed.
I also ended up becoming a bespoke VxWorks admin because the guy who volunteered was never available, and someone asserted my code wasn't working well on VxWorks, so of course I had to know enough to do benchmarks. I fixed a few problems but the real issue was the hard drive wasn't doing DMA due to the kernel not recognizing the processor revision number.
Somewhere between those two events I realized that if my part of the project is great but the whole project is on fire, nobody cares about my stuff. I don't get points for being right and the team being wrong. I mean, I do for some people, but I still feel bad at the end of the project, and those are the 'points' I have to live with the most.
The way that played on on the latter project is that once we got our shit together, we started work stealing from our peer orgs. Volunteering to carve off little pieces of interface between us and 'take care' of this bit of data handling here and that one over there. I came to realize that the Org subconsciously knew this and any past success they had was due to self-organization and collective work-stealing.
Throw people at it, turn a blind eye to precise mission statements, and hope for the best.
At smaller companies the title tends to be only a title and many people are basically full-stack and know a bit of everything. And you're trading hats to get jobs done.
If you truly enjoy this part of the job, you can go for a freelance career - build a network and reputation as the person you call to solve the hard stuff, and do mostly that, no leetcode interviews required...
Anyway, it sounds like you've worked in some really shitty companies / shitty people.
The problem the term 'devops' is addressing, the way I understand it, isn't that of insufficient individual jack-of-tradesness, but that of too much separation between (internal) organisations. It should be perfectly possible to build a "devops" team from deeply specialized experts and turning it into a job description seems quite a stretch to me.
But you might actually want some jack-of-trades types nonetheless, because those isolated organizations weren't completely without merit: they are good at solving that problem affectionately called the bus factor. A single ops guy in a devops team is a single point of failure and the mitigation preparation of keeping some of the not-so-ops peers sufficiently in the loop will be much more dependent on organic motivation than the counterpart in a specialist company branch. There, substitutability would be much easier to ensure with formal process.
It doesn’t matter as long as communication is open.
Proper ops is all about reducing the bus factor. Procedures don’t depend on people, they depend on being good at communicating.
If your ops person got hit by a car, I would expect that everything would continue running, I would also expect to find some clear documentation, this is mandatory.
We have a way to prostitute words in this field... agile, QA, devops.
jan [at] competition [dot] company
If you want more information on the project we're working on the site is https://rennsport.gg
We're building a hard-core racing simulator game with a backend which can persist car ownership in a way that feels authentic. (IE; not just tied to a game).
I am interviewing right now and my previous experience includes devops roles and SRE roles, so I get contacted for both by recruiters. After hearing about the responsibilities and examples for these roles I can only come to the conclusion that titles are a waste of time. Even the "level" of a title of Staff, Principal, Lead, etc are a waste of time. Just call the job what it is and if you can't decide on what it is at least to an 80% level, maybe you're asking too much for that job?
I do work on Cloud Infrastructure at times, but honestly it's the smallest part of what I do. It's usually addressed in the architectural designs and I have to touch it incrementally. What I do much more often is writing tools, daemons, and services for distributed systems. I end up calling myself a Distributed Systems Software Engineer to reflect the idea that what I work on is systems and software, and most of them are non-monoliths (from a systems perspective).
Do you have any thoughts on Systems Engineers or the title that I prefer to call myself?
There are people who might refer to you as a backend programmer (if the focus is services and daemons), or platform engineer (if the focus is developer velocity).
Largely it depends what your primary focus is.
And more importantly, the backend engineers should be encouraged to be grateful for the PRs (even if it’s not acceptable for some reason). The more eyes on the code, the better.
I think developers naturally tend to specialise (at least for a time) but ultimately we all need to understand and contribute to code regardless if it’s front end, backend, build, test, …
DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways.
All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be.
There is also a lot of rewriting of history that gets thrown in, similar to how when people talk about cloud then the only alternative is to start making CPUs by hand and begin building your own nuclear reactors. It’s the idea of what came before, not the reality, that people seem to be defensive of.
It’s honestly exhausting to discuss.
So instead I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.
Roles so simple that you already have a clue what they do, which is sort of the point of job titles.