IMHO most disasters I’ve seen occur when ‘Product’ tries to strictly control the ‘Developers’. It just does not work for certain kinds of projects. It also diminishes the developers’ involvment and leads to a culture of mediocrity and CYAs.
Also, a person that does not have to do the work will always be waaaay more optimistic about how much it takes to do the work.
How do you fix it? Trust and ownership. Product and Dev need to have an interface - a boundry if you will - and the discussions and talk are about defining this interface and setting expectations. Product trusts Dev on implementation. Dev owns the delivery and makes it happen.
Almost every issue I've had with the business/developer dichotomy (and I use that instead of product because different orgs place product differently) has been due to the business trying to pitch solutions to dev that invariably end up not actually addressing the problem (even simple things like "that's the wrong color" or similar, are because they believe they're responsible for the solution, and they have a very clear idea of what the solution should be, without actually being able to articulate or recognize the problem. Does the color actually matter? If you have a problem that implies it does, yes, if you just came up with a solution on your own and in your head that solution was yellow, no).
The real world is a messy place where even with a good team, a good plan and sensible expectations you can fail spectacularly.
If there is one thing that I really like is aggressive validation of the assumptions that are being made + keeping the product as small and lean as humanly possible (and the latter should apply across the board to everything you do).
Complexity and Entropy are preparing the coffin for your work even before you have the idea. You can give ‘em a run for their money, but in most cases you’re going to end up 6 feet under. As soon as you accept this and are genuinely open to failure you will be ready for the opposite.
Currently having issues with this one right now, and it is the most frustrating thing I've experienced at a job.
Software is hard because we 'don't know the unknowns'.
It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.
If we knew how to write the module, we'd just do it, test it, and voila.
But most of our time is spent 'figuring stuff out'. Oh, product has asked for a tiny change (support a specific kind of font), but our libraries don't support that, meaning major possible architectural changes.
The other major unspoken factor is that requirements change. In fact, this is the #1 thing, really. Requirements change because customer's needs change, because business changes because our understanding of the market changes, etc., and this is a reality of our world. So the nature of how we're going to adapt is difficult to predict.
This is why the #1 predictor of outcomes is the quality of developers on the team i.e. experience, intelligence, know-how of the domain.
If you're doing something where there are very few unknowns, well then it should be possible to schedule within reason.
Walking on water and developing software from a specification are easy - if both are frozen - Edward Berard
Comparably it'd be as if building developer built from prefab that changes the design every week.
I hear that it doesn't work that well for buildings either, as you end up with people in the field detecting bugs in the plans and having to improvise fixes without input from the designers.
Soil and dirt samples at $100K - just to do a bid on a project, let alone the actual build.
Can't spend $500M on something only to find you have to tear it down.
Software is not magic, but it does require thoughtfulness and experience ... (hint to all those who think startups require mostly new/young blood - it's mostly wrong, but that's for another post)
I have never seen that work in practice.
1. Are you sure you can do that in half a day?
2. You forgot about this one part of the task that you thought would take half a day.
3. There was a bug in the plugin that you were going to use to do this task in half a day, and you spent three days fixing it.
4. This task will actually take longer, but management won't buy that. Looks like it should be half a day on paper, so that's what we'll say.
5. I've never even done this one thing before, but it sounds like it should just take a couple of hours.
I understand the concept - trying to say something will take a week is even more error prone. But I think most of us are still pretty bad at estimating a budget for tasks even at such a small scale.
In practice, being able to break things down meaningfully and accurately to such a minute level implies that a lot of time was spent in grooming, which means we're just back to waterfall again. (Or the tasks at hand really are just that simple and/or the people involved have a lot of expertise.)
Not to mention that that approach only works if you produce another version of something you already produced many times. There certainly are such development efforts, for example, I think anything data- and form based, like data entry + database + some reporting, what used to be paper cards a long time ago, and then punch cars, those type of apps can probably be made using that method.
So that's when they had to build the freight version of the Boeing 747. Then they thought "that was easy", and started the Dreamliner, something very different that they had not done before... (and it even still is an airplane, so not even that outrageously different as different pieces of software can be)
I think the "break it down" approach also faces the problem of the "coastline length measurement".
I'm also reminded of an anecdote from my early days with computers: I had to write an assembler routine (as part of an 8 bit assembler program) that seemed too hard, so I outsourced it to someone else. He didn't know what to do either, so I explained it to him. When I was done I realized I had just implemented that difficult function - I wrote it down in half an hour and that was that. Planning well, finding out what to do, is harder than the actual programming. Again and again, in all projects, I find that the >90% part is finding out what to do. There are stories from projects that lost all their code, years of work, and they thought they were done and finished (business kaputt) - but when they rewrote the software they fond it took them only weeks or instead of years. When you already know what you want and how the end result is supposed to look like (and feel and behave) then writing it in code is the easy part.
So you've advocating an approach where you... don't plan? I'm confused.
Perhaps you're getting too hung up on the half day part? That measure is a useful forcing function that ensures the dev team has thought through the approach in enough detail to identify where the risks are and what smart trade-offs can be made.
We practice a lean-laced Scrum with 1-week sprints. And yes, it works. And yes, we go fast. So yes, it can work (thought we have several years of refining our methodology and have a big culture of pragmatism and problem-solving).
Indeed, for a lot of us in smaller companies, getting our clients to actually consider taking an agile approach at all is a stretch.
Story generation is done by the PO, that's most of his job (though we split stories during backlog refinement if needed).
I don't know if it sounds like a lot (I feel like it could), but turns out it's actually efficient, every minute of each meeting is productive because we know exactly what we're going in for, how it will go and what you want to get out of it!
You're identifying what the outcome of good planning looks like, not how it's done. What prevents this is that teams read "agile" as "make it up as you go along," when in reality if you're spending more than 10 minutes discussing a feature, make planning that feature a sprint task.
> In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back.
It's not generally the team in that room discussing those deadlines, though, it's software development manager and product manager. The worst managers are software engineers who get promoted to management, and when they get into those meetings they look at their shoes and are railroaded by upper management.
You really do need a manager who has the intestinal fortitude to go to bat for you.
Re #1, this is kinda the system 2 vs 1 thing in the article. I was lucky enough to have a PM drill me into this. More specifically: The bigger a task is, the likelier it's going to go over time. The bigger a task is, also the less likely each bit is well specified and more likely that there are significant portions glossed over that you'll discover during implementation, which will delay things and/or result in them being done poorly. If you're having a hard time breaking the task down, it's probably because the task isn't well defined enough, so go back to stakeholders and define better. In this model, planning takes a lot of time and effort. Also in this model, research-y tasks are different, and instead get a fixed timebox after which you have to re-decide how much more time to spend on it if at all.
Re #2, part of the problem here I think is that PMs aren't trusted / empowered with that ability or final say. "Stakeholders" get to make that decision, but with ignorance of what it's currently like in the trenches.
I'll add this: a product team that ships earns 1) more credibility, and 2) that credibility is reciprocated towards the team in the form of greater flexibility and freedom to set their own targets.
In my experience, they just pretend it doesn't exist and remove it from the "plan". Where I see this the most is in figuring out how things work. For some reason, non-programmers have it in their head that once you've finished college, you ought to know (even though they don't) how everything that goes on a computer works instantaneously. We're going to use Apache Kafka! Ok, need to spend some time figuring out how that works. Um, ok, that should take like an hour, right? Because there's no time in the budget to learn anything. If you push back, they say, "ok, never use anything new that you have to learn. Only use the things you already know". That doesn't work either, because the things that we have to integrate with don't work with the old things. Or, "you don't need to know how that works, Bob knows how that works, just ask him. But don't let it take any time, because there's no time in the plan for that" And Bob is sick of all your questions that you'd know the answer to if you'd just spent a two or three days reading the online documentation and installing a test system and testing it out.
Most of that will be doing spikes: http://agiledictionary.com/209/spike/
It's one the most underappreciated agile concepts. I wish people would shut up about standups once in a while and give more thought to doing spikes.
Spending 30 minutes researching four different libraries to see which one to would be a spike, for instance, but it doesn't necessarily involve building a proof of concept.
[1] https://www.tamingdata.com/wp-content/uploads/2010/07/tree-s...
And then some poor soul gets to rework all this...
Basic abstractions help a lot. Specifically, having things weakly coupled.
What does work is waterfall. Whatever is in the repo at the end gets shipped.
To work it takes being comfortable saying "that's my/our fault", and my experience is that a lot of people who would rather die than admit fault. It's part of a larger raft of skills around transparency. So they cover up problems hoping for some miracle that never comes (or more cynically, they hope to be employed somewhere else before the shit hits the fan).
If the client or management want to change the scope, in any way, the discussion must immediately start with questions such as what other feature are we dropping, how much time are we adding to the delivery plan, what are the consequences on other features or the current design and code?
Even deciding to remove a feature can cause delays (e.g. having to re-structure a test scenario). Or changing a string in the user interface, which may require re-arranging the layout, re-translating the string, suddenly having to manage plurals, checking that a variation of the old string wasn't already used elsewhere, etc.
The original primary point of the Agile Manifesto is to stop doing that. There's no excuse for it, unless you are required by law as a government contractor, and in thar case no one expects a successful result anyway.
Deadlines presume a known fixed scope. But I’ve yet to meet a CEO or sales team or whomever gets to decide what features go in who will fix the scope if a project.
They are always willing to add features, but can never take them out.
But you are right that estimates are for managers. Managers should watch their reporter's progress and use that as evidence to build evidence. If the first task of 20 took 1 week, it's a 2year project. Update estimate as time and progress and work remaining increases.
I’m quite serious.
The thing you need to realize is that you’ll get it wrong, no matter the tool... until you invest time in reviewing past estimates and plans.
People think because they’re engineers they can just magically make estimates... but its a skill. If you don’t have a feedback loop to measure your progress and adjust your actions, you’ll literally never improve at it.
...that’s why even ‘Senior’ developers and ‘technical leads’ are often terrible at this; they’ve simply never bothered to seriously make an effort to learn how to do it.
1. Poor engineering planning. The way to do this well is to break down a feature until they’re at about half-day sizes tasks. Identify any obvious risks or ambiguities with the technical approach and communicate them to Product and figure out how to de-risk them. Communicate costs to Product so PMs can make cost/benefit trade offs (eg “if we built it the way you specced it it’ll take two weeks. But if you make these trade offs it’ll take two days.” As a Product Manager I might not want the feature if it costs two weeks but do if it costs less than 5 days. Without costing the features in dev days I can’t makr this trade off).
2. Failure to make trade offs that drive towards shipping software. The way you ship is by declaring it’s done, even when it’s not really done. When code is being written unexpected issues always arise. It’s Product’s job to make hard trade offs that drive towards shipping. Eg “that’s an edge case bug, let’s punt it to v2” Or “let’s cut that nice to have feature and squash the showstopper bugs and ship”.
People bring up impossible deadlines set by management. In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back. And if a deadline can’t be moved, it’s all the more important to know how much features will cost and make hard trade offs to get the product out the door.