And then came the Assumptions.
And the Assumptions were without form.
And darkness was upon the face of the Workers.
And they spoke among themselves, saying, "It is a crock of shit, and it stinketh."
And the workers went unto their Supervisors and said, "It is a pail of dung, and none may abide the odour thereof."
And the Supervisors went unto their Managers, saying, "It is a container of excrement, and it is very strong, such that none may abide by it."
And the Managers went unto their Directors, saying, "It is a vessel of fertiliser, and none may abide its strength."
And the Directors spoke amongst themselves, saying one to another, "It contains that which aids plant growth, and it is very strong."
And the Directors then went onto the Vice Presidents, saying unto them, "It promotes growth and is very powerful."
And the Vice Presidents went unto the President, saying unto him, "This new plan will actively promote the growth and vigour of the company; with powerful effects."
And the President looked upon the Plan, and saw that it was good.
And the Plan became Policy.
This is How Shit Happens.
Engineers hate releasing poor quality products, but the reality is that customers often will buy and use something laden with defects rather than nothing at all. And in any case, after the initial botched release, you can always fix stuff. See Apple.
Here's the reality, and maybe you've forgotten: The engineers are the ones who are going to get dumped on, and you and the VP's are the ones who are going to reap the benefits. Never forget that.
Not that I'm claiming that all executive decisions are clueless, most people are just trying to do a good job with limited information, and mistakes happen. But you run into enough executives without respect that runs in the other direction, and you start to question all deadlines.
(Which isn't necessarily a bad thing. In a healthy environment, questioning the feasibility of a deadline can lead to appropriate descoping of overreaching features, reduction of risk, etc. But the situations that lead to this kind of suspicion are rarely healthy to begin with.)
This is everything that is wrong with software.
What is software really? It is translation of business process and need into code - I spent the first 10 years of my work life mastering code, I will spend the rest of it understanding whatever business I'm working for.
Assuming you have some experienced engineers (not mangers) on your staff then having the vp set a date (and defacto choose the project) is absolutely backwards.
Bring the problem to your engineers (money) and don't look for a moon shot look for 10 smaller projects that get you 1/2 way to the goal in 1/10th the time. Don't think of it as a "hack day" think of it as a hack week or hack month.
If you have a large project and it is to "save the company" (or push it forward) and your treating it like a must do rather than a foundation (and taking care) your creating debt and starting a cycle that you will never pay off.
This is the same type of argument that makes short-term profit take precedence over long-term success. Why in the world are we getting a deadline of 4 months? You, as an executive, had to know burn rate was blowing through cash a lot earlier than that.
Engineers hate releasing poor quality products because we know we'll eventually have to fix them and we will likely be given almost no leeway time-wise to fix the actual debt, as time will always favor ugly hacks that "work" over well-written code that prevents the problem in the future.
The tech titans aren't monoliths with a collective hive-mind. Each business unit and team is responsible for projecting revenue targets and then hitting them, same as any other kind of business. And, yes, if the business unit managers fail to hit their financial goals too many times, their project will be cut and the team reassigned or laid off, same as a startup running out of financial runway.
Sometimes...
Other times its driven by the fact they are personally going to get a fat bonus
What worked for us is what's basically mentioned in the article: taking a more structured, slightly painstaking approach to estimation. It's more tedious than gut feel story point estimation, but a lot less painful towards the end of the project.
Most dev work right now is still just attempting to deliver correct software. let alone on time or on budget.
House building has become somewhat more complicated over the years as building codes are made stricter and new techniques are developed, but for the most part it is just a switch to a new way of doing things vs. in the software world where we will continue to pile on more and more levels of abstraction, but can't reasonably expect to have at least some kind of understanding of the underlying layers.
"Debuggers" are the monitoring, telemetry, sensing, and forensics equipment, data, and processes applied to each test, launch, and component.
We have debuggers because code is otherwise a black box. Debuggers are a visualisation technology.
I worked professionally in a couple of high-tech engineering jobs, as architect, surveyor, city planner, civil engineering, construction, stage design, movies, automotive engineering (F1), games and at internet service providers. (but no airplanes, nuclear power plants or rockets, sorry. Lots of hospitals still.) The SW development jobs compared to that were always the easiest, most creative, most rewarding, and best paid also.
The problem with SW planning is pure lack of experience and poor engineering practices. A good SW engineer can hold his estimates, just as an experienced architect can hold his date and costs. In practice it rarely happens, I only know a couple of successful projects with smaller companies, but the good and big companies mostly deliver on time and budget.
As you mentioned houses and bridges. With houses you don't change engineering paradigms every 5 years as it might fancy you. You never try out new materials which are not tested over 10 years at least. These things need to stay up for 50 years at least. With bridges you cannot really plan them properly. Well, you can. But a typical bridge engineer multiplies the safety factor with 10, leading up to 10x more material as needed, as you can hardly estimate the dynamic peaks and resonance frequencies. With highly dynamic forces such as e.g. in a F1 gearbox, a huge spring with max 4000 Nm forces on your shafts. There you cannot apply any 10x rule. This thing needs to be perfect. You plan to avoid the peaks at all, avoiding the dangerous frequencies. Comparable to wind forces on a bridge. But in practice you look at best practices. You build dynamic physical models and simulations. And measure. And apply safety margins. E.g. with the big steel bridge next to my home, the famous "Blaues Wunder"in Dresden, the general public and politicians never trusted their engineers with this fancy new steel design, so it had to made 10x bigger and heavier, and even on opening day they invited the whole town to test it live, with lots of heavy trucks, a tramway and pedestrians walking over the bridge and jumping on it all at once. It stills stands today, but could have been made much simpler and elegant by todays knowledge.
Now compare that to modern SW engineering. You got a fancy new framework and language, looking for engineers with at least 10 years experience in that, with the framework existing for 2 years. You got massively overhyped tools which rarely deliver on their hype, complete lack of security, memory safety, type safety or concurrency safety, completely outdated API's like blocking IO or POSIX or threads, and undesigned monsters like C or C++. Good luck with that. That's of course a death march.
Still with proper tools and proper management it's much easier than everything else. In automotive everything was properly planned, with traditional waterfall. Everything worked as planned. Even if it was hugely complex, much more complex than a simple OS. In my own SW I never miss my milestones or overdo my budget. In architecture we always did. With huge costs up into billions. Only once we went under costs by 200 millions with the biggest building construction project in Austria. This was a miracle then.
We need to set realistic goals so that we can work hard to meet them. The mindset you are describing is part of the reasons that we have these problems in the first place.
I've worked with Engineering VP's/CTO's that are pressured by deadlines and promise unrealistic features. The key element needed in this is a 'leader' that can push back and manage expectations.
Generally that results in one of two things:
* Product delivered on-time with massive technical debt.
* Product delivered late with massive technical debt.
Frankly I don't know if adding front-line engineers to the deadline decisions is going to make the issue better or worse, but fundamentally having non-technical or formerly-technical people defining deadlines definitely doesn't work.