And herein lies the rub. If a project is more than 6 months to a year in duration, there is very little chance that what the customer now wants is what they wanted before. Requirements aren't created instantly on day 1 ready for dev, you could easily have several years of requirements, during which time people change, the world changes, the law changes, priority changes and then what?
Even systems that are relatively unchanging like the air traffic control system they built in the UK still had a whole raft of issues that needed addressing and at this point, the documentation becomes out-of-date and changes are extortionately more expensive to resolve.
Depends largely on the customer. Just do anything defence, related to aircraft, or industrial control systems, more often than not, requirements are set in stone, and modifications require approval and consequent fees.
Now, those industries throw a lot of money at actual R&D to know what requirements can be requested, which is very different from your usual software shop, where "R&D" is some brief talk during refinement or at most a dedicated user story to look into the subject.
The customer might claim something different, but for a couple of large and failing projects in industrial automation I've had to fight hard to be able to iterate and rescue them from the jaws of defeat.
You've had different experience?
If you stray outside that, both waterfall and agile struggle… but waterfall has the potential fatal flaw of delivering a product no one wants (or doesn’t work).
This is why agile was introduced. There were software products being made that ended up just not working… so people saw that problem and tried to fix it with agile. But agile just fixes the “don’t deliver a product that doesn’t work” problem.
Agile techniques both surface those errors early, so they're course-corrected quickly, and provide a set of clear rules for the rest of the organisation to abide by which should mean that the delivery team can't get overloaded into a deathmarch.
"You're doing it wrong" is not a compelling argument.
A competent, motivated, invested, and empowered team don't need any particular methodology to succeed. Methodologies are adopted precisely because many people don't do things properly much of the time. A methodology which doesn't account for this and doesn't self-correct when done improperly isn't worth a whole lot.
Waterfall done properly produces great results. Agile done properly produces great results. Both of them produce bad results when done poorly. Proponents of one tend to compare their methodology done properly to alternative methodologies done poorly. That's a pointless conversation to have.
Of course you can also waste too much time on planning and information gathering, but that's not something I have ever witnessed. Usually time is wasted by starting to develop without making a proper plan.
That's how I usually work (in personal projects). I wonder what it would take to convince management to do the same. In theory it should "just" duplicate the budget of any given project (which doesn't sound totally wrong).
It sometimes happens to me, though, that I need to reach v3 in order to go back to v1. I just feel this way of doing software to be the most natural way.
Build the redo time into your estimates.
Non-technical management has no concept of what's involved in building software. They don't need or particularly want to know. They care about things being "done." You don't need to convince them of anything, you just need to consistently and reliably show progress in a way that they can comprehend.
It's very irritating to see no progress for a month, then see a demo of something that appears to do everything you asked for, then hear that it won't be "ready" (whatever that means) for another month because the whole thing needs to be redone.
It's very reassuring to see demos every two weeks, each one showing an obviously incomplete product, but with clear progress between each demo culminating in a completed and delightful product after two months.
The actual development process for the two scenarios above can be exactly the same, it's just how they're messaged.
This is "managing up."
The teacher drew a very simple chart : y = t. "This is, in a given project lasting from t=0 to t=1, the amount of practical information that you have about how to design this project. At t=1.0, you have 100% of the information. When you start, you have about zero information about it, just guesses.
Then another line : y = 1-t. "And this, is the amount of design freedom you have during the project. At the beginning, everything is possible but at the end, big changes can not be made."
"This is what we call the curse of project management."
It has really be enlightening. Make prototypes, be courageous enough to scrape entire designs, and when you have the resources for it : make a total reset for version 2.0 and redo v 1.0 correctly.
Of course, this is highly subject dependent. You don't build a bridge the way you build an experimental robot, but this explained well the interest of non-waterfall models.