If your business leaders are prone to changing their minds at the drop of a hat, you probably aren't going to actually change that. Putting process in place to make it painful to do that is just going to cause pain for the people who have the decision imposed upon them - the CEO's just going to tell other people to figure out how to make it work.
Is there a way to bottom-up build something focused instead on reducing that pain from those pivots? Just-in-time planning whenever possible, for instance, to minimize wasted time invested in would-be future projects?
The timeframe matters, here, though. Are the wild changes once a quarter? Once a week? Once a year?
I spend a couple of years at a startup I co-founded; requirements volatility was legitimately very high. We were very experimental, which meant we learned a lot, which meant frequent wild changes. We had a long-term vision, and plenty of theories about the future, but rarely planned more than a couple of weeks out.
This was honestly fine. The product manager (my cofounder) was really good at including us on the necessity of the changes, so it never felt irrational or out of control. Our process was highly tuned for minimum waste. And we were militant about a lot of quality-oriented practices, including unit and acceptance tests, pair programming, paying down technical debt, continuous deployment, frequent releases, and close collaboration with the product manager. That meant that however things changed, we were prepared.
I think if you can do it at a start-up that's shipping every few hours, it definitely works at much lower levels of requirements volatility. To me the real issue at scale is that if agency moves away from the team level and gets concentrated at the executive level, then decisions at least appear irrational, and often are because real-world information becomes ever more removed from the decision makers.
Now, every decision (new or changed) has a cost to be implemented. Some more than others. If you have decision-makers who refuse to recognize that, and think the amount of work they can ask a given staff with a given level of skill to accomplish in a set time period is unbounded -- no process is gonna save you from misery.
I work on 3-12 month projects in an agency, we don't really do Scrum. It's mostly some version of waterfall, and occasionally dual-track agile. Generally the process of determining what to build is squeezed into the cracks. I just recently built about 40% of a prototype for a SaaS in a couple weeks with fractional team input, and then had to lay out the rest of the flows in a week. After that, more needs popped up (I can only imagine the shitshow that development will be). This is the worst case scenario.
In less-worse cases, it seems like this becomes an issue of how to compare a requested change's value to its cost. Without enough data, this can be really hard to actually know well enough to have a grounded position on. It seems like consensus here is to focus on the cost of the change, rather than the potential value.
It also seems like there's little faith in managers. I don't totally know what it's like out there for ya'll though.
Yes, exactly. I know some good people who are dedicated Scrum fans. In their view, the engineering practices are all implied, because if you do have a relatively autonomous team dedicated to continuous improvement, you'll get there. And it works, because they are smart and dedicated people committed to doing the right thing.
So in that sense, the idea of Scrum is a good idea. But Scrum-the-movement and Scrum-the-business have fallen very short of that ideal. So in practice about 98% of Scrum implementations I've seen make me despair for humanity.
> What I really wish Scrum or Agile had, was a more unbreakable way to let development teams say no
One interesting version of that was the Extreme Programming Bill of Rights. One version of it is here (and in XP jargon "customer" is something like "product manager":
http://www.agilenutshell.com/bill_of_rights
It's not perfect, but I think it was a good attempt to make clear that there's a useful balance of power.
Once we got approval for this, it meant that product owner had to focus on communicating to us why the business had certain priorities and desired certain pivots, instead of just nuking our desired plan of work and superseding it "from on high."
It meant everyone had to trust us as the team that we wanted the business to succeed! (Who'd have thought!) And we weren't going to act childish and just demand backlog items and sprint work that satisfied our preferences for some fun project or some back-burner idea. That, as grownups, we would actually consider when business priorities made sense to supersede our intended plans, and act accordingly.
After a while, we exercised our privilege a few times to ignore or decline "urgent" work that the product owner relayed to us. Instead we invested in much needed refactoring stories, fleshed out more serious design documentations, researched detailed trade-offs between approaches or third party tools we would need in the near future.
Over about a year's time span, this worked incredibly well. The team felt happier and more productive. There was less tension with the product owner since he would not circumvent our plans and dictate new priorities all the time. And people learned they could trust us to understand the underlying business considerations and deploy our team accordingly.
In our company, this was very much as "un-Agile" as it gets. It was the farthest thing from what "Agile" methods meant in the company as there could be, and in large part we only got permission to work this way because our output was in a very specialized domain area that had really critical requirements for some large business initiatives. Most teams could not have possibly negotiated this arrangement.
Like clockwork though, the director of our business unit was surprisingly fired and replaced with no explanation or warning, and the new director stopped us from doing this at that point. It reverted to the usual thing where sprint planning is just a Wheel of Fortune gameshow where you learn the radically new and unplanned pivots you'll be forced to make for the next two weeks. Like trying to win a 100m race by dancing on a DDR mat at the starting line.
Anyway, long story short, the barrier to anti-quality principle was that it's totally the team's discretion as to what work is admitted into the backlog. If the team ignores legit business considerations at their own peril, to goof around with fun stuff, then the team owns that failure. But no matter what, they are not told by the product owner what goes in the backlog, and it must be a sacred right for the team to tell the product owner "no" when management proposes to add work or change direction in a clearly stupid way.
If management doesn't buy in and I can operate my own low-overhead, low-formalism workflow, then so be it. I suffer the badness of management, but I don't also pay time waster costs for a formalized process whose sole purpose of better facilitating productivity is categorically disallowed in the first place.
I personally, as a matter of experience and opinion, would take it further to say that certain properties of Agile / Scrum actually add fuel to the fire and allow management's already bad behavior to manifest in even worse ways. But even if someone doesn't agree with that, I think the point stands: why incur the overhead costs of Agile if the same outcomes happen because of mismanagement anyway?
I had a less extreme version that still served our needs. My rule was that a fixed percentage of effort, say 80%, would go to business priorities. The product manager could certainly ask about where the time was going, but they couldn't claim it unilaterally. The rest of it could be ordered and reordered as the product manager thought best. We in turn could ask about that.
This worked well for us, in that it let us keep the code clean and make tooling improvements without having to explain to a non-developer what was going on.
It was also helpful that we were good at expressing technical-sounding things in user terms. E.g., all scaling and reliability work was put into the backlog with titles like, "system handles 1000 simultaneous users" or "system survives web server failure". And a lot of research was covered by the product manager buying information rather than features.
Of course, this was at a startup, where there was no "on high". The product manager was the CEO, so in theory he could have fired us, but mainly we just collaborated on getting shit done.
I think this is a key insight, and I suspect for many people (myself included), the phrase you use when you say "Low-end Scrum" is just the only type of widespread Scrum we've ever seen practiced anywhere, ever, to the point that it makes us question whether Scrum itself is partly responsible, even in an idealized format.
One of my good friends came up with the idea he called "barriers to anti-quality" to describe this. A good workflow process establishes inviolable norms that act like preventative barriers to doing things in an anti-quality way -- so that you specifically arrange the world such that trying to take a quality shortcut actually costs you more than just doing it in the quality way.
We proposed trying to judge workflow and management practices based on how willing upper management was to give up control of the process in exchange for erecting barriers to anti-quality.
If they wanted to force a team to pivot in some quality-destructive way, just on their whimsical preferences for reprioritizing different tasks according to vague business needs, they could not do it.
Barriers to anti-quality would be like Odysseus pre-committing to be tied to the mast, so then when the Siren song comes, he cannot give in the temptation.
I think a lot of the spirit of early Agile was meant to suggest something like this. Even though the method talks about being flexible and adapting when there are real business needs to adapt to, this gets bastardized when middle and upper management conflate adapting to real business priorities, and begrudgingly paying a cost in terms of quality lost due to pivoting developers, with demanding that people have to adapt to the management flavor of the month, unsubstantiated ideas about business priorities, often stemming from political power struggles, restructurings, quarterly earnings, etc. etc.
What I really wish Scrum or Agile had, was a more unbreakable way to let development teams say no, and decline work or decline pivots. Basically to veto changes made to the backlog, and say that it violates an anti-quality principle for no good reason.
A key failing of modern Agile / Scrum is that it doesn't create any type of stop-increasing-our-entropy-with-unsubstantiated-management-preferences notion of a barrier to anti-quality.
Don't get me wrong, this is hardly a failing that us unique to Agile / Scrum. But I think when people talk about why in principle these methods fail, it's that they lack any mechanism of enforcement for this. As a result, they can be so easily politically subverted to just be a pivoting, bludgeoning tool of misguided or unexplained management choices, that almost always trade-off quality for anything else, and instead of ever looking back to understand when an under-investment in quality actually cost them, they just explain it away as some other team's fault.
In a sense, Agile / Scrum (or 'Low-end Scrum' as you call it), is a tool to create managerial Dutch books. A manager gets to unilaterally disrupt on-going investment in quality without ever being held accountable when that gamble on short-term anti-quality causes losses.
By the end, it makes developers very cranky that they are being jerked around with all this prescriptive and time-wasting overhead of meetings, planning, estimating, etc., only for this political anti-quality nonsense to go uncorrected anyway. If that's going to happen anyway, then what's the supposed value of all that costly Agile-specific overhead?