It depends, the idea of working in short sprints with deliverables at the end of it is that often you do not know what you need and the best way of finding out is by trying out and talking to the customer.
The place I'm currently working at is agile on paper but 90% of the work is just implementing external requirements we have no say in. Needless to say, not analysing the problem up front and not designing the general architecture up front leads to building roadblocks later on that prevent getting work done.
To stay with your metaphor: Sometimes you have to eat shit (because of legal requirements, etc.). Planning ahead is very useful in those situations for not choking on the shit.
If I design a machine, and I need a gear or a cam or even a more complex component, I can send the specifications for that part to a machinist who knows nothing about the "big picture" of what I am building. Yet he can make that part of it to perfection. Does this lead to the same sort of problems with the development of physical systems?
I would argue that physical systems aren’t developed in a “waterfall” method.
In mechanical design classes in engineering school, we learned to start with low fidelity sketches to capture customer intent. We come up with several options, build those into increasingly higher fidelity 3D models, use simulation to refine, take a few candidates and get physical prototypes, do physical testing (strength, endurance, integration, etc.), determine the best one, then go into limited production to prove out and establish the production process, then go into full production.
We are taught (and in the automotive industry it’s a requirement) to have cross functional teams involved in the design stages of both the item and the manufacturing process.
To your point about inadequate spec: My opinion is there are so many different backgrounds coming into software that there is no common language or background.
I think everyone thinks their way is better and it’s hard to communicate technical ideas when you need to constantly recreate and translate between terminology and documentation methods. This lack of convergence and common knowledge is what I think results in poor specs.
So software development are all about writing down the specifications precisely enough for the compiler to create the artifact. Mixing up this process with manufacturing is a source of many of the problems in software development.
Instead of using the term waterfall as the counterpoint of agile I prefer to call it the human centipede model. In this model all of the vision, creativity, and flexibility stays with the head and the rest of the centipede just eats their shit. Developers can't see further than the next person's ass and have no idea why they are actually building the functional specifications that are fed to them. Implementation becomes completely disconnected from design which leads to compromised quality, missed deadlines, and products that miss the mark.
No task management framework is going to solve these problems.