Then I make more promises and estimates, playing the emotional card of "sunk cost" to convince upper management to continue funding the team rather than torpedo the project. Then we ship, and I quickly get everyone on my team promoted "because we shipped" before it becomes clear to upper management that we shipped something essentially worthless to the company. Because we're in a huge profitable company, everyone can jump ship to the next shiny thing, and whoever is left holding the potato on its way to deprecation just gets re-org'd when it gets canned.
I know it's a bullshit game. But I play it well, and all the software developers on my team get paid and promoted because I am so good at it.
You sound wise and such in general, but perhaps your time would be better spent making a difference in how people believe work should be done at a place that actually allows brewing better practices instead of lauding bad ones.
Will you then argue that all new projects should avoid deadline dates, that no project should have anything other than an MVP and a iteration plan after that?
How do we change the expectations?
This is exactly the point of the article: Organizations routinely fail to prevent untenable projects from moving forward.
Why not work on something so valuable that even if it takes twice as long as the best estimate, with twice the headcount working on it, it would still be worth it?
As a a manager, I feel that helping to find projects that have that amount of value is an important part of my job, so that if I am fighting for a project my team's working on, I'm actually in the right from a business case point of view, not simply playing some BS political game.
If there are no projects that fit that value filter, then that's a completely different organizational problem.
On one level I have to take my hat off to you because you sound like exactly the kind of manager - one who has your team's back - that a lot of people would want to work for.
On another level you're part of the problem: you're wasting organisational resources that could be better and more profitably used elsewhere.
Still, needs must I suppose, because the real fault here - as the article itself hints - is in the determination of what the most valuable use of everyone's time is. I.e., if projects are going to be late (and they are) then you ought only to deploy teams to work on the ones with a putative 10x ROI.
But there was one occasion, where a product launch had been announced by a C*O before the product was finished, and the engineers were very clear that this launch date was impossible. So there was a lot of drama and bad feelings, and the engineering team eventually said, "okay, let's make a detailed estimate so we can be certain". The estimate came in at three weeks development time, one week after the announced launch date.
One week of the estimate was meant for testing and fixing problems that are found in testing. The business experts immediately jumped on this: "Testing and fixing unplanned errors won't be a big deal and certainly not a whole week. We'll skip that so we can get it done in two weeks".
Thankfully, this is retail banking, so releasing an obviously broken product flies even less than missing the deadline. The product was eventually launched uneventfully, one week after the detailed estimate and two weeks after the announced launch date. All in all a success.
The biggest pet peeve is when I give an estimate such as "I don't currently see a reason why we can't deliver X by the end of next week, assuming no other priorities come up." Of course something more important comes up in the mean time and I get scolded in a large meeting with a statement such as "You promised me I'd have X by Friday!" — this is especially infuriating when it is the PM's other project that is the higher priority interruption.
In short it sounds like you don't have the power to say no - but actually you do.
Of course, if you already have a toxic development culture like the one you describe, then they'd probably abuse more agile processes too.
On the reverse of course it makes sense to have estimates otherwise how should managers or product people decide what should be done next?
A big part of the issue is we don't have good knowledge how to estimate work. It's not a skill that is heavily invested in and is wrongly assumed as an inherent ability.