If I can assume you’re an SDE for a moment, I actually agree with the part of you not providing estimates. Recently I’ve done the initial estimate for projects exclusively between SDM and PMT with no engineers involved. It has been a lot more reliable and it seems that SDEs are very happy to be absolved of this responsibility. This does require SDMs and PMTs with a good amount of experience.
Not a mature engineering discipline.
If you can budget a planning phase in development that allows you to quickly explore the unknown unknowns and known unknowns to investigate critical bottlenecks and uncertainty before estimating and you're able to lock that down with a set of features, then I think you can create decent estimates.
That's rarely how any development environment in current existence operates though, at least from my anecdata. Most are 'agile' that can drastically shift directions, feature/scope creep is a continuous problem, there's a constant time pressure exerted by managenent on developmeny teams in hope to optimize a bit more productivity out of their high price tags which gives no slack space for them to dig into these issues (except maybe some personal time).
The entire modern development culture in most business environments is designed in a way that makes any sort of good quality estimation nearly impossible. In the best of conditions it can be hard but manageable, most environments are the worst of conditions.
How many projects in mature engineering disciplines are accurately estimated? I get the sense that this is a general problem, even outside of software.
The concept of standardized parts and assembly lines is less than a century old. How accurate do you think their estimates were before they figured out the basic principles of repeatability?
The "mature" engineering disciplines literally just punted on the problem for several centuries, only giving birth to systems engineering [1] in the mid 20th century because they were so bad at it and everyone's back was against the wall in WWII. Before it became its own recognized field, project management in engineering was worse than it is in software now.
Not coincidentally Bell Labs - the company that basically kick started the computing industry - was also the biggest player in the formalization of systems engineering. Since then its been adapted as the methodology for managing engineering projects by everyone from civil engineers to NASA [2]. Any estimate you see for a nontrivial project from the past half century isn't the result of mechanical, civil, or electrical engineers but the product of systems engineers.
[1] https://en.wikipedia.org/wiki/Systems_engineering
[2] https://www.nasa.gov/connect/ebooks/nasa-systems-engineering...
There are lots of reasons for that, which can fall outside of the scope of engineering, but the same is true for software.
My favorite moments from events like WWDC are when you are introduced to some really cool feature or app for the first time, and then the speaker goes "available today". The fans love it, the news sites love it, whenever you can immediately try out something the hype for that product goes up 10x.
When you only show the product when it's finished, you no longer need to estimate anything.
> dependent activities
This is where I think the real problem is. If you have a feature that no one else depends on, if you have a story that nothing else depends on, don't estimate it. It doesn't matter. That is a nice to have. It'll arrive in some sprint eventually.
If you have a feature that other things depend on, before estimating it, you should ask if it is possible to create that feature without any dependencies. Could the other team that you are working with code such that they work with your current product, and when they update and you update, the new feature turns on? Can you do the same for their product? Great, we don't need to depend on each-others updates.
If we had a mature engineering system, I don't think we would ever have any dependencies.
Perhaps you aren't perfect, and there is no way to rid the dependency. Go ahead, estimate it. Then double that estimation. Then convert that estimation into one larger unit. 1 day becomes 2 weeks. 2 weeks becomes 4 months. There, now you can build schedules around it.
If the team provides an estimate that falls one month after the deadline, the pressure for them to change their estimate will be extreme as well. In reality, if management has already decided when the product should be released, they don't give a fuck about an estimate. What they're interested in is for the team to take ownership of a decision they didn't make, and that's what the estimate is for.
One of the things I dislike when I hear this is that it says nothing about the difficulty or cost of getting the estimates. Yes, good estimates are extremely valuable, but solving the halting problem would also be very valuable. That doesn't mean it's going to happen.
A big issue is that to get good estimates, often we need to solve most of the hard parts of the problem. How do we account for the time needed to get the estimates?
> If we don’t estimate then we are saying that software engineering is not an engineering discipline.
I'm not sure I believe this. There are plenty of non-software engineering projects that are late and go over budget. It wouldn't surprise me if that was the norm. Certainly with construction projects it happens all the time.
I'm actually curious about which engineering disciplines actually come up with good estimates. When developing a new type of airplane, or a new engine, are estimates typically accurate? It seems unlikely to me.
The halting problem is mostly a non-problem in settings that really need a proof. We have non-turing-complete languages that let us produce programs that provably halt.
That they are not mainstream tends to show that we don't really need that proof very often actually.
If you can't avoid giving estimation, try to pad it as much as possible, add every single uncertainty to the task list, and estimate very conservative. Add enough time for testing, communication, work on change requests.
And even after the original estimate is approved / published be sure to communicate updates to the estimation after every single change request, question, bug, new insight.
And if something by chance take less time than estimated, make sure not to decrease estimation, but to use it as a buffer.