I don't know why you'd blame ops for the crappyness of the tools they have at their disposal.
Yes, Ansible, Salt, Puppet, and Chef are spaghetti-code inducing congealed messes of design. So are large collections of complex shell scripts.
So what's the alternative? What spherical cow of a configuration management tool from Platonic dev heaven shall be foisted on us this time?
I'm sure it'll be super clean and elegant this time, unlike the last thousand shitty tools they made.
And don't get me started on devs that think they're qualified to do ops when all they know is their language of choice (if even that) and have never thought about the network, security, capacity, redundancy, failover, reliabililty, hardware, backups, the rest of the company or other users.
More seriously it always going to be complicated and annoying. It's really past time we started dealing with the fundamental complexity of everything we are trying to do with software.
The problem comes from the management/business side. They hire devs and tell them that ship features as fast as you can. Also they hire ops guys and tell them that I want this whole thing super reliable, we can’t afford a minute downtime.
In my opinion this is why DevOps is mostly pointless. We are trying to fix with tooling, processes, new tech, and fancy roles the fact that business people don’t want to make compromises or choose between the pace of delivery and reliability.
OP’s should replace QA at that table to rebalance the equation. But again, and as you illustrated, we have an adversarial relationship that takes a lot of across the aisle work to introduce sanity.
even if you could staff a competent group, they are often left with nothing to do while the devs 'work their magic'. and suddenly its 3 months past the original 'functionally complete' deadline, and QA is given 1 week to do what they .. oh, maybe 3 days really, to do what they need to do.
when it works its irreplaceable. when it doesn't its just a lot of noise and spend and flailing.
I know why it’s gone, I’m just not happy about it.
The QA people hated it though.
Terraform and Ansible look like gyroscopes compared to the build process of any modern software stack. We offered our dev teams a whole ass pizza party every time they had 10 green builds (on main) in a row. In three years we've paid it out once.
Ops people are not used to thinking in boundary conditions. Hell, devs forget half the time. That’s part of why people wanted to merge them in the first place. Get the right sorts of cynicism together in a room and make me something with a big green button an idiot can push while everyone is in a meeting.
Nothing demotivates people like a half assed reward. If you want something never to get done, offer a bad carrot. If you want only one thing to get done and nothing ever again, offer a good reward but then make the recipient fight you to get it. I thought pizza parties were the worst, and then I got an award that was announced but not delivered, and spent almost eight hours over the next month poking people to give me my goddamned prize. It became a second job and at the end I ended up resenting the entire experience.
Later at another job when we in theory had a generous reimbursement process that was actually a ton of red tape, I had flashbacks to that incident.
Also don't assume we don't know our coworkers, when they first got to 8 in a row the slack channel looked like the Twitch chat of a popular streamer.
We have the buy-in from management that if it happens every day then it happens every day.
We are planning to change up the food if it ever happens again, pizza party is just a good title
They historically took on the reliability role, of nobody else did, but they were implementing reliability on top of a house of cards, which is a kind of hypocrisy that makes even mediocre devs bristle. Don’t lecture me on robust software, boyo. Your tools are made of string cheese and staples.