First is the "gap" between those doing the work, and those writing the checks. (When it's the same person, this problem disappears.)
The guy editing the checks likes to understand progress is being made, and that the project both has an end and will be successfully completed.
The second problem is that by it's nature software "never ends" and many (dare I say most?) projects fail and are simply abandoned.
The moment the check writer is not the direct manager of the development you have an intractable problem. The person in-between (quite literally middle management), is often not technical. But he has to convince the bean-counters that this project is "on time and on budget".
He can't help but feel sometimes that he's herding cats. He's an irritant to those who are "doing the work" so they treat interactions with him as a waste of time. Inevitably he starts trying to measure things. (And we all know what that means.)
His job is hard. He's stuck between developers who don't want anything to do with him and higher-ups who want reassurance, bit don't really trust what he's saying.
The miracle is not that this process sometimes fails. The miracle is that it ever works at all.
And sure, you may not like your meetings, but at least understanding the game might help you understand why his job is the crappiest of all of them.
Managers with no domain knowledge are just an extra layer of indirection and overhead without any of the benefits. Would any IC want their career gated by someone who has zero insight into the team's day to day duties, doubly so if it's at a supposed "tech" company?
At the moment I'm having fun developing again, but I have had experience of management as well.
Obviously my experience does not speak to all cases, and I'm fortunate that most middle managers I've dealt with are competant.
I am in the fortunate position of being able to "speak truth to power" though, and generally been able to dispassionate translate technobabble into managerspeak, and help all the layers understand each other better.
I've found that when both sides understand (as much as they able) the complex demands being encountered, more realistic expectations and demands are made (and usually achieved.)
But I've been lucky. I've had good dev teams, good middle management, and good product owners. Who have all been pulling together to make projects successful. When one group is pulling the other way, things get tougher.
Well, sure, but that's what the daily standup is for. So that the problem solvers on the team can understand the process, the problems it faces, and, importantly, find ways to optimize it.
But it seems, for how crappy you claim it to be, those in that position don't want to talk about it in fear that it will get optimized away. Job protectionism trumps all, I suppose.
In my experience, with 2 week sprints, the business doesn't really have to think about anything outside of bite sized chunks or even how those bite sized chunks affect later bite sized chunks. They can make a snap decision without thinking about it because it can be rectified in the next sprint. Almost no-one has a "big picture," view of the software.
Before agile, we essentially did 4 releases a month (this was back on the burn a CD days), 2 major releases and 2 minor patch releases. It worked really well and we didn't get the burnout as much by sprinting a marathon. With sprints, I always feel burned out. No sense of accomplishment, no letup, working hard just gets you more work. It's like laying bricks one at a time on a wall with unlimited length.
Also feeling bad every 2 weeks because planning was really bad and now you need to say you're sorry the task has to roll over the next sprint and your manager has to explain to their manager, etc, etc.
Product Management has fallen a tremendous distance from where it used to be. PMs who have come up completely under Scrum seem to think stories and epics barely require more than 1-2 sentences and the developers should have to figure everything out from there.
Even senior people in their early 40s have completely forgotten how to specify or document anything well.
It makes me wish I worked on something that involved physical products. The whole thing with agile/scrum is that management can change their mind at any time for any reason and the process gives them justification. It works in pure software, it doesn't work in anything physical because they had to sign the POs to buy materials and parts.
Correction, 4 releases a year, or one every 3 months.
Just to call it out: Having CI/CD doesn't mean you have to do all these things, or do scrum. It's imho just good engineering practice. I have it in my side projects, and I've had it at work on a Kanban-driven team (as well as on scrum).
It removes disputes over build process (eg, nobody can ever say "well, it complies on my machine"). It is a form of build/deploy documentation. It removes bottlenecks ("only Tom knows how to deploy that service, but he's on vacation") and stupid mistakes ("whoever deployed this last did it wrong and left a bunch of old files around").
I also have found deploying regularly is stress-reducing. For one, with CD the development environment is running the same steps, and we know it works because we do it dozens of times a week. The longer it's been since last prod deploy, the less confident we can be it'll go smoothly, whereas when it gets deployed every few days it becomes a non-event.
When something breaks, having only 1 or 3 changes makes it really easy to figure out why. Recovering after a big release with 40 PRs in it is absolutely painful.
None of this means you have to release everything on a regular schedule (like a sprint). You can still do long running branches or feature-flag things off, Just try not to go too long without deploying something.
This is, by far, the most important reason to practice continuous deployment. I've been part of enough of these fire fighting sessions following big releases to see that it's not a sustainable way to deploy software. And yet, I've never been able to convince any boss I've ever had to adopt CD because they're worried it'll introduce more regressions into production.
It's one reason I'm really thankful to work in a certified/regulated industry. It's mandated to effectively prove your work to get the sign off. And since you can't really sell too well in the area without that cert, it's a big part of development.
But holy hell is it still hard to get specific people to test their work. They just don't even think about it.
We often would go days without having a formal meeting.
But, software was different then as well. Everything is interconnected now. One team dropping the ball affects countless other teams. Deployments were whenever we felt a new one was due or at a multi month cadence. Did this introduce problems, yes, but at the same time, they came in controlled release cycles.
Nothing against CI/CD and continuous delivery, but the hamster wheel has gotten to a point where we have to release all the time. Corners are cut on everything, and testing is given lip service.
At this point, I am a manager, and SCRUM stresses me out. Either let us work from a queue, or give us a project with a deadline. Give me back a stupid gant or pert chart. At least then it was, is this done to allow XYZ, not we are going to accomplish this in the next 2 (arbitrary) weeks.
So much more to say, but I am toast.