Agree about the line worker thing too, and that I think is a fundamental problem with most methodologies. Great authors write great books. Great programmers write great software. Any methodology that pretends there is a way around that I think is bound to fail.
When I embark on a new project, it's routine for me to spend a couple of weeks with the customer sounding out their requirements and helping them form an initial vision of what they want the software to do. (Big companies call this "business analysis".)
Once I'm reasonably certain what sort of thing the customer wants, I start work on the first "sprint". This first sprint is often quite a bit longer than subsequent sprints, and incorporates the holistic design phase you mention. It's not uncommon for me to spend a month designing and working on this first iteration.
The goal of this first iteration is to capture the essence of the vision without implementing the details. Only once the customer has viewed the results do I reduce the time for each sprint to a week. (I believe two weeks is more common, but this period suits me and seems to fit with what my customers want too.)
A dozen or so iterations/sprints later, and I tend to be done. At least for a while, anyway... inevitably a customer will decide they want another big feature and I start over with the business analysis and the long first iteration followed by shorter incremental steps.
Agile is born from rapid prototyping and makes a lot of sense for that environment. If you're doing work where one day you might be developing one thing, and the next day you might be working on a completely different product altogether, then reducing the cost of change is the only thing that makes sense. If you're not, then don't use pure agile, it doesn't make that much sense.
There's obviously an issue with one size fits all approaches to agile, and as agile matures people realize that more and more, instead of forcing you to do everything by the book. The point is, as you get more and more experience with driving software projects, you need to be smart enough to understand what will make sense and what won't.
Agile methods were born from in-house enterprise software projects. The ur-project for Extreme Programming was a payroll system, and SCRUM's was something similarly dull. The original SCRUM paper says, "Scrum is concerned with the management, enhancement and maintenance of an existing product[...]. Scrum is not concerned with new or re-engineered systems development efforts."
If I'm doing rapid prototyping, that's when I'm least interested in Agile techniques like pair programming, test-driven development, and refactoring. Those only really pay off in the long haul.
If you are lucky enough to work in a domain where you can reliably plan years or decades ahead, then you can still happily use Agile methods. Incremental delivery still lets you get more for your money by putting the most valuable parts of the system in production first.
But if, like most people, you have competitors, or if you learn new things when you put your product in the hands of users, then the ability to respond to change is absolutely vital. In that case, holistic reasoning is an ongoing process, not something that happened once, years ago.
Some tasks require lots of planning. When you are building a complex system, you cannot plan as you go, and in some cases your first usable release cannot happen after the first 'sprint'. In this way, agile programming assumes that the programmer cannot make long term development plans, and in fact prevents him from doing so. While this eliminates some of the risk, it also prevents programmers from reasoning about their problem holistically, and can deliver inferior or non functioning solutions for projects that do not happen to fit the model well.
This refusal to allow programmers to reason holistically is part of what us critics mean when we say that agile attempts to turn programmers into line workers.
Complex projects, like building a jetliner (or the programming equivalent), cannot be "continuously integrated" with no up front planning. For projects like this, you need to decide the thrust of your engines before you start to design the airframe, or your entire project may fall apart, literally.