The triangle is real[1]. Something's gotta give when a deadline can't move, and as we know from Brooks, software development speed doesn't increase linearly with more people. So it's primarily quality or scope that must suffer to hit a software deadline.
[1] - https://en.wikipedia.org/wiki/Project_management_triangle
Honestly the OS X itself (uhhh macOS now) is still fine, even with Big Sur and its weird iOS controls, however the other software is sometimes weird; but - let's say Numbers was never good and iWeb was always downright bad. And iTunes was bad in Jobs era and is still bad (as Music).
I however still kind of miss when native Cocoa apps were sort of good and not just ported iOS app via Catalyst or something. But whatever. I'm old I guess
this really is key, if you can do it
alot of companies though, are "business driven" which means releases and schedules are set beforehand, which makes this hard... (but if you can do it, its great!)
And no doubt a great deal of crunch time, their fair share of delays, some major messes and the occasional outright cancellation. And if I'm not wrong their roadmap is not public information.
I personally find it hard to believe that a large company can exhibit all those adjectives you list. Everyone finds this hard.
Do they tell you at the beginning of that cycle what features to expect? Or do they constantly work on features, and then at presentation time do a freeze?
Requirements gathering, general communication, testing, production rollout, coordination, research, final approval - it all adds up. For a 2 week project you might only actually spend a day of ass-in-seat coding. And that ass-in-seat coding is what they tend to estimate.
That's what the research has previously found[0] decades ago -- engineers will generally correctly estimate the part they are best at, but will either neglect or under-estimate other work. The old-school ways of dealing with this are canned Work Breakdown Structures and checklists.
This is not unique to software.
[0] van Genuchten, "Why is Software Late? An Empirical Study of Reasons For Delay in Software Development" https://web.archive.org/web/20160914040151/https://lab.cs.ru...
For stuff that I'm doing on my own? I can estimate pretty well. I know more or less how long the writing will take based on length. I'll allocate more time for research if I need to do more than a base amount. And then I add some slack in case I'm struggling with something, priority interrupts happen, or I'm just not into it on some given day. (Underpromise. Overdeliver.) It only gets harder with bigger projects. It helps that I'm quite fast compared to most people so even with significantly underpromising people are usually very happy with the estimate.
But where it's really hard is work that involves getting buy-in and input from a lot of different people. Those always take forever. (And at this point I pretty much know they'll take forever, a lot of which is just latency rather than actual active work.)
Estimation is really hard, especially if you're dealing with new challenges, new people, and new expectations. Throwing moving targets (unplanned changes) into the mix, as you noticed, makes estimation even more difficult.
The thing is, it very rarely is the case that a wrong estimate will sink a project. Things get delivered late all the time. No one dies. Most stakeholders aren't like two year olds throwing a tantrum because an unforeseen problem delays a deliverable by a few days (admittedly some are close).
The actual definition of failure also needs to be considered. Is it considered a failure because a project manager ended a colored gantt bar at a particular date and the actual time was longer? Or is it a failure because the customer cancelled the contract because a deliverable was not available on the agreed-upon date? Or is it something far more nebulous, like a slow death march towards a misguided goal that burns everyone out and compels them to sip artisanal coffee and daydream about leaving instead of grinding onwards and hitting targets?
Ultimately, people "don't care" because accurate estimates are not the most critical problem on their mind. Most of the time an accurate answer to "When will it [the deliverable] be done?" has about as much gravitas as guessing the number of jellybeans in a jar at a kid's birthday party.
Doing a retrospective and analyzing what went wrong when a project delivery slipped expectations is certainly reasonable but it so easily slips into a blame storm and/or people making commitments that aren't realistic. Getting straight answers from folks for this kind of analysis requires a level of psychological safety that just isn't present in most workplaces.
There is a story that Microsoft Word was a staggering failure in terms of actual time and cost to implement versus its estimates.
It then went on to be among the most lucrative ($ billions) pieces of software ever created.
Was Microsoft Word a failure?
Any kind of reality doesn't factor into it.
In the Word example, somebody really thought this project was worthwhile for some reason, so they gave a very low estimate to get things started. As soon as money has been spent on a project, it isn't likely to be cancelled as easily, so you are safe to continue even if you exceed your original estimate. In the worst case, you need a few excuses.
External estimates, i.e. estimates you use for writing up a contract with an external customer, are different. Those actually do matter, depending on the wording of the contract. But a smart contractor will try to get the contract into a shape such that the above for the internal estimate does apply.
For everyone (n00b to senior) I would always recommend designing & breaking things down into no smaller 1/2 day blocks, where nothing went longer than 3 days (for inexperienced), but longer for more experienced. Senior devs should be able to be able to be trusted (to be accurate) for week-long blocks.
Use a Gannt chart (ms project or similar) and monitor the inexperienced, gauging their real-life times with their estimates, adjusting accordingly. Give semi regular feedback at the accuracy so that they (& you, the manager) are working with.
In one of my companies, the most inaccurate we were for any project was that we were late 2 days over 2 years. Build on trust, build on meetings and other time sinks. Include testing, sign offs, etc.
So, absolute n00bs need close mentoring so that the (senior) mentor can begin to gauge the growth rate... Which will be steep... But different for everyone.
You can get very high accuracy on estimates by spending lots of time creating smaller stories and encouraging people to over estimate. This may not be in the best interest of the project or the business.
For example say you have a CRUD application with a page for editing and searching complex objects. Rather than building the entire search page with all of its filters, and then building the edit page with all of its fields, you build a simplified version of the search page with only a few filters and a simplified version of the edit page with fields that correspond to those filters. You also make sure the functionality integrates with the backend to form a cohesive, vertical feature that can go through QA, and possibly the client. Once that is working you expand the list of fields and filters.
This approach tends to surface design issues much faster, and is more adaptable to change requests from the client.
another commenter sort of hinted at this, but this type of process has a downside of prioritizing short term design decisions that lead to working code. It takes time and sometimes a lot of thought to think about the system and the problems deeply and ask yourself if you've created the right primitives.
Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work.
I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV
Only problem: the customer might not like what he gets, even if it works exactly as planned.
Ah, but the customer does not know everything they want, due to the fractal nature of reality. The closer you get to the end product, the more detail is resolved, and more work is added.
Xeno's paradox [0] doesn't apply here. Some projects do actually finish.
[0] https://en.wikipedia.org/wiki/Zeno%27s_paradoxes#Achilles_an...
Your time estimates should always include time for research and design. The customer can sign off on what you're about to write at the end of that (whereas beforehand you give them a wild ballpark figure made by senior devs - not management!)
You're right, design is very very necessary and should be given plenty of time. But it's not a magic bullet.
If they want the liberty to "change their mind on the fly" then they should buy man-days which is a completely other thing. Or, they can buy supplementary mandays for whatever they want at the daily rate without changing the contract, or, they can just wait to finish this contract and start another one.
A programmer can accurately estimate the schedule for only the repeated and the redundant. Yet,
A programmer's job is to automate the repeated and the redundant. Thus,
A programmer delivering to an estimated or predictable schedule is...
Not doing their job (or is redundant).
I don't think having the detailed requirements made estimates much more accurate. However it made creating those estimates take an order of magnitude longer - developers needed much much more time to absorb all the details.
At that point I decided I only ever want to work with senior developers who have good imagination and can give reasonable triplets (solution, # of devs to do it, approximate time) for requested pair (idea, quality level).
Then you need to train the IT professionals to build the dynamic nature into the conversation and process. You dont quote a number, you give a range. You communicate confidence levels and uncertainty clearly and continuously. You update assumptions as new information comes available. You _push back_ when the uncertainty is frowned on.
It can be done successfully on a micro scale sure, by working on it within an organization and/or team. But its not scalable and its not a 5-step plan. Its hard.
When people ask me how can they learn, I tell them to find old project schedules and other project documents. Looking at past projects makes the learning more objective, and it is not happening under time pressure.
My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.)
But at that point nobody cares! People go on with further projects, and no data is ever collected.
Nobody learns.
When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away.
Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done.
It's insane.