The real problem with agile (even well-done agile) is the interface with non-agile upper management. Sooner or later you usually run into someone who wants to see your detailed project development plan and timeline, or some such. Answering "Why would we need one of those? We just talk to each other" does not go well, even if it's true that you actually don't need one of those because you've addressed the problems the document is supposed to address simply by regular informal communication. (Been there, done that.)

If upper management doesn't "get" agile, at some layer you have to transition to non-agile. There's an impedance mismatch at that layer. If someone doesn't actively translate things into upper-management terms, they can decide to kill the project, simply because they can't tell what's going on.