Some folks tend to think that if a plan is produced but not followed exactly, then it wasn’t worth producing the plan. But actually having produced the plan is sometimes what makes it easy to deviate when complications arise. If a new decision is made that affects the plan, everyone can see exactly what the compromises are and be on the same page.

I always despise these kinds of exercises.

We know how to roll up estimates correctly with statistics--even moreso if we have historical data.

The problem is that management always hates the accurate number.

And that's where the dysfunction sets in.

While everybody else rolls things up "normally", I almost always roll up an "accurate" timeline and try to inform people up front. It rarely wins me friends, but it sometimes protects my team when things are going haywire later.

Few hypotheses are necessary for a thought experiment that reproduces the issue.

Given a project:

* some people/teams produce accurate estimates

* some people/teams produce overly optimistic estimates

If decision-maker is not aware, it can easily conflate optimistic estimate with a more efficient team (or just more enticing).

If the possibility of overly optimistic estimates are not taken seriously, the more accurate team can seem just less efficient / lazy / trying to charge more for the same work / etc. For example if the decision-maker is actually a client comparing several suppliers.

Extra risk if decision-making is diluted among many people (harder to get clear and consistent view).

This puts the teams in a competition similar to the prisoner's dilemma. Honest estimates get punished, treacherous estimates are favored. No wonder so many projects are over-budget, when they are not too over-budget to complete at all.

The only actual hypothesis here is: "decision-maker conflating optimistic estimates with better value". It seems very common. If decision-maker doesn't fall into this trap, the problem disappears, right?

What can we do to shift the equilibrium? A manifesto?

The "always" part isn't true in my experience. In fact, it's not even mostly true for me. Maybe I've just been lucky.

As an Eng. Leader I use planning and estimates to communicate with other stakeholders.

One thing the majority of people fail at is giving variance when providing a forecast/estimate.

If someone asks me to get an quick estimate for developing something I'll give them a range with wide margin, around the avg time I think it will take. If they want to reduce that range, then I need info to understand the requirements better.

Problem has always been that what engineers provide as an ESTIMATE, business takes as a deadline. So, engineers: always add buffer to your estimate.

It makes everyone focus on what is most important.

Are those features really necessary for the release? Can we change the requirements to accomplish the goal easier? Is it important enough that we should delay it?

Without a deadline, engineers waste time working on stuff that is fun and interesting to them and not the hard stuff that is needed to ship a product across the finish line.

I do that along with every other engineer I know. It is human nature.

Anyway, if you are using sprints, there actually are due dates. It’s just there are a lot of them occurring in rapid succession, and we don’t commit to what we’ll finish in the following due date until after the impending one.

Why is the threat of a deadline required to ship product?

We do it at the start, people change their mind 10 times and then we deliver something different and it's considered business as usual.

But if I suggest just once that we cut the estimates out, people get angry and afraid.

As if they all WANT to be lied at...

well... I mean, that's the whole point - end game is to have engineer-sourced figures that you can show to your banker so that they put cash in your account and become subject to the opportunity cost paradox when it's too late and you have to move your deadlines.

You tell people the truth and they call you unprofessional for it.

In my experience hard deadlines are a bad idea but estimating work is also necessary to allow the organization to plan what to build. It also creates a beneficial cycle where stakeholders will reduce the feature set because typically smaller projects get front-loaded in the development roadmap to deliver value faster

Nowhere in the comment you responded to is there an implication that deadlines are “hard” (can’t be changed or have consequences for missing them). That’s for one simple reason. Everybody knows they aren’t. NASA isn’t going to launch a vehicle that they know isn’t ready. Video games, hardware, and movies get delayed. Our software project can have features cut or the release date changed. And, for the most part, it’s okay.

That deadlines are “soft” in some sense doesn’t undercut any of the substantive points made against them, nor does it actually evidence any of the claims made for them.

The distinction between hard and soft deadlines is not relevant to my argument.

Still management definitely feels something has gone wrong when you say at least half the projects will be over estimate.