Lots of retroactive changing of gantt charts, too :)

OK, but then how do you know what each prototype is required to do?

It’s true that you might be lucky enough to work on a project where your employer/client/customer is willing to let you make some of the decisions. In that case, you are effectively defining some of the requirements yourself and hoping that your employer/client/customer will be happy with your decisions. That’s a risk you can mutually agree to take, and sometimes it will pay off royally, and sometimes it will end in disaster, and probably most of the time it will end somewhere in between.

However, in my experience, the more important the software you’re building is, the less likely it is that you will have that kind of flexibility for the developers. At the extreme end of the industry, if you’re building something like the control system for medical equipment that will kill the patient if it goes wrong or a billion dollar satellite that you get one shot at launching, you probably aren’t going to be doing it through informal discussions with a “product owner” and relying on TDD to make sure everything is working (where “working” in that case is really a euphemism for “whatever test conditions an individual developer felt might matter”).

I have yet to see a single project in all my decades of experience in this industry where the requirements didn't change during the project.

Of course, but every time the requirements change, you still need to identify what the change is so you can change your implementation accordingly. There are many ways you can address that problem, some of which involve one big requirements spec document and some of which don’t, but if you don’t end up with some clear and well-maintained spec each time, you don’t know what you’re trying to build and you have no meaningful definition of “working software”.

For most projects, both "knowing what to build" and "working software" will never be 100% clear.

You define an area of exploration for each prototype. Like "we think that the hard part of this problem is getting the encryption right, so we're going to explore that in this prototype".

And I thought SpaceX disproved this whole waterfall approach to rocket science? They have used iterative prototypes to build better rockets considerably faster than the waterfall approach used by the rest of the industry.

My argument is that whatever stage you’re at in your development process, if you don’t have clear requirements, you don’t know what you’re trying to build and you have no way to define whether your software is working or not. The comparison from the Agile Manifesto makes little sense, because having comprehensive documentation of your requirements and acceptance criteria is a prerequisite for even being able to define the working software that the Manifesto says is more valuable.

Of course you might do all kinds of prototypes and proofs-of-concept to help you explore the problem space and clarify the requirements in the early stages of your project, and maybe again later if there are significant changes. But this work isn’t about building working software, because you don’t know what that means yet; it’s about doing experiments to help you find out.

Providing the customer with a working prototype to criticise often (always) allows them to refine their vision of what they wanted and helps them communicate what they need. It's always easier to criticise something than create - saying "the interaction around X doesn't feel right, can we make it cornflower blue?" is a lot easier than "the X interaction must be three shades of grey off cornflower blue" in a spec document.

That's been my experience, anyway. Insisting on a comprehensive spec at the start has always been more effort than its worth, because no-one knows what they actually want at the start.

In the case you’re talking about, creating a prototype and then soliciting feedback from the client might be an excellent way to go. More generally, building a project in stages with regular interaction with and feedback from the customer or other stakeholders is often helpful. My point is that you still need to decide what you’re putting in that prototype or any of those earlier stages, because ultimately some developer has to write that code and management need to tell them what to create.

So you might well have a spec, in whatever form, that has initially has gaps or ambiguities relative to the final product you’ll want at the end of the whole process. That’s fine, and as you say, it is very common. But it should still reflect everything you know you want at the current time.

If and when you gain new information, such as feedback from your client after showing them a prototype, you can update your requirements to fill in the gaps or clarify the ambiguities in light of the additional knowledge you now have. You still have a clear record of what is needed, taking into account all the actual decisions that have been made and all the actual feedback you’ve received, and that is (to the best of your knowledge at that time) what defines whether or not you currently have working software.

Put another way, if you don’t maintain a good record of that information, you’re just building your software based on hearsay. Some of that information about what the customer/stakeholder actually wants will inevitably be lost or distorted with the passage of time and perhaps the changing make-up of your team. And then you definitely don’t have working software, whether or not you realise it.