But that view of requirements is not borne out by the reality for most projects. You're presuming that it's possible to gather the "ideal requirements" when the project first starts, with enough due diligence - and also, that those requirements are fixed.

In my current job, we're producing a service that has to court a handful of very large clients. Even if there was a well-defined idea of what the service should eventually look like in 5 years, a lot of feedback is required to discover how it should look now. Which client needs more attention? Where is the biggest opportunity for additional value? How are users actually using the service, ignoring what they said they were going to do with it?

That last part is the most important - requirements are in reality a feedback process for which the existing product is one input. You cannot analyse how users will empirically interact with a product that does not yet exist. Abstract analysis is no substitute for data.

What your service looks like now and what it looks like in five years are obviously two different questions, but a proper analysis will divide the issue between now and 5 years from now and come up with requirements that fill the gaps. This doesn't mean things get set in stone and aren't adaptable to changing needs - when this condition is identified, the manger/developer need only apply the workflow again, and revise the specifications with the updated data, and a new development plan can be formulated. Maybe this is 'agile', but again - it speaks to the fact that waterfall is a naturally occurring phenomenon in engineering/technical matters, and thus should be applied consistently, completely, in order to provide fruitful results.

> You cannot analyse how users will empirically interact with a product that does not yet exist.

I don't agree with this, as I believe it is very, very glib. You can of course empirically interact, by wearing the user hat. Too often the developer/manager/user hats are considered adversarial - but when the decision is made to be flexible about which of these hats one is wearing, during analysis, makes all the difference between whether your product is successful or not. Software is a social service - rigid developers who cannot put on the user hat, are not providing the most intrinsic aspect of that service in their field.

>You're presuming that it's possible to gather the "ideal requirements" when the project first starts, with enough due diligence - and also, that those requirements are fixed.

I make the claim that this ideal can be attained, by ensuring that the early steps in the waterfall process are actually applied. You are correct in noting that "when you don't do things right, right things don't happen", however ..

The issue with the Waterfall model is that this approach doesn't work. Aiming for "now" means you come out with an outdated product in the future. Aiming for "5 years from now" means speculating on what users will eventually want, which is very error-prone. Trying to adjust course midway through is a nightmare - and completely defeats the point of trying to get requirements "right" the first time.

>You can of course empirically interact, by wearing the user hat.

This is not empirical: it does not involve observation or measurement of the real world. Speculating about user behaviour is no substitute for concrete data about how users actually behave.

All the more reason not to rely on the proper wearing of a user hat (how are you going to know, by the way?), but actually work with the users instead and spend time creating the necessary artefacts to capture their perspective as soon and as well as possible.

Yes, fine. So there is no (true Scotsman) domain expert. Grandparent's point is also just that actually building the application and seeing how users interact with it is a valid method of analysis either in a prototype phase or actual evolving production software.

I want to highlight this, because it's almost universally true on any given project that there are no complete experts. But I'll be looking at domain not as a business domain, but actually considering if you can find a domain expert in the technology stack a project is planned in.

When requirements are written up, there is never a domain expert on all the technologies to be used that is so experienced in the technology that they will not get things wrong. Software development is so intertwined and so fast-paced that we are at the mercy of tech stack we use too.

Imagine you are well-versed in Postgres, but your new project requires you to use Postgres FTS extension. Without learning the ins-and-outs of Postgres FTS extension, you'll make grave estimation and "planning" mistakes. And then you are supposed to deploy on RDS instead, where there is another set of considerations. And this being a new project, you are asked to use asyncio psycopg for the first time, so there will be more gotchas as well.

Basically, the number of "tools" we shift between is so vast that nobody can be an expert in them. The rate of change is huge as well. Just count all the Kubernetes tools today, and imagine having to use any one of a dozen of mini-k8s set-ups for local development. Just exploring the tech stack is a full time job for a dozen people, and you wouldn't be getting anything done. While others in the market would.

So, if you are in a world where you've got your tech stack pre-set, you've got plenty of experience with it, and you are a domain expert on the business problem you are looking to solve, sure: waterfall-like approach will not put you at a disadvantage. But introduce any one of the newer components (cloud scale-out), and an approach to gain expertise first will have you run-down by the competition.

Having been in the field for some time, and reading HN and talking with people in the field for as long, I believe your general attitude is one of someone who has an idea for how things should work in a stable, ideal world with unchanging toolsets and unchanging technologies. This is not the world of technology.

> I don't agree with this, as I believe it is very, very glib

This implies that all A/B testing is worthless, which is .. surprising.

You can certainly refine the software over time (A/B testing, if you will) to more closely attain the ideal, which you may not have well defined at the beginning of the project if you don't perform an adequate review of the needs of the user.

But you can certainly also complete a user analysis that produces requirements, which when fulfilled, solve the problem in its entirety for the user. These two extremes are not absolute - sometimes, if the analysis is incomplete, A/B testing can rescue the project regardless of how poorly the first analysis was performed - but A/B testing, it could be argued, is applying waterfall properly: iteratively, as intended... but by all means, call it 'agile' if that makes the difference to the team involved.

Can you specify which book/process/document(s) you are using in your day-to-day? This could be very interesting in future discussions with clients, to say the least!

Also to note, the paper is titled "Managing the Development of Large Software Systems."

The author outlines 5 steps they believe need to be used to improve the success of a project. One step is called "do it twice" in which the author advocates the development of a prototype well in advance of delivering the final product. I think this is iteration. It's not indefinite iteration, but it's definitely not showing the big bang single release waterfall approach either.

It also advocates to involve the customer throughout the process instead of only at the end.

The summary says to refer to Figure 10 as the process that incorporates the author's steps. This final diagram is pretty far from the simplified and flawed waterfall picture that has been pulled from the front of the document.

[1] http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970...

Waterfall is not iterative. That’s the whole point. You go over the waterfall once. If you’re iterating then, by definition, it’s not waterfall.

> you can certainly also complete a user analysis that produces requirements, which when fulfilled, solve the problem in its entirety for the user

This is almost always false, for any non-trivial project, within the usual commercial constraints of money and time.

Not really. It indicates a general inability to predict the future.

That's really the core of the issue. Requirements analysis can be done badly but even when done well it doesnt make you good at predicting the future.

The longer the feedback loop the more the ground changes under your feet.

>Pushing the problem to or away- from Managers is just one way of saying "I'm too lazy/incompetent to adequately complete the Analysis phase

I used to believe precisely this in my early 20s.

My "aha" moment came when I built software solely for myself and even then I realized that this didnt stop the requirements from changing because of surprises that kept being thrown my way and problems that I uncovered only in retrospect from actually using the software.

For sure not all problem spaces are this chaotic but most software problem spaces are.

That's the same kind of problem with waterfall. You want them to do the analysis perfectly, and then you want the world to not move while the implementation happens. Neither of those is possible. And then you blame the analysts for failing to do the impossible.

The analysis phase is never complete enough. The conditions are never the same by the time the implementation is complete. That's just reality. What are you going to do about it? You need to steer during implementation, not just once at the beginning.

Relevant clip perfectly highlighting the problems you have during requirement analysis: https://www.youtube.com/watch?v=BKorP55Aqvg

No, we're suggesting the analysis/specification phase should be comprised of what are, effectively, many really small waterfalls. And that any effort to obtain specs for an entire system without first building some minimal version of the system is doomed to failure.

For a small enough feature, yes, you end up doing waterfall. The problem is breaking up a massive system into those small enough pieces.