I believe that is the crux of the issue. Larger organizations, and especially those that view IT as a cost center, do not feel comfortable giving control to programmers.
Every time a consultant proposes a new method with a new name, it seems like it's just a way to create a side industry; a long con that doesn't really attempt to fix the larger issue. I believe the hard problem to solve (which may be impossible, I'm not sure) is getting an organization to see their programmers as business collaborators rather than an inconvenient necessity.
This. Craft and passion beat any process. These people will organize themselves into the most efficient process eventually.
My direct experience of working with programmers over the years is that this will generally descend rapidly into an inefficient mess. Yes – maybe it works, sometimes. I'd argue that's the exception, rather than the rule.
If you work with people who dont care, youre screwed. Nothing will ever work to force them to actively engage with the process of creating good software.
..but, surprisingly, when you do work with people who do care, they self organize remarkably well. The Valve megacorporation is fundamentally based on this. It works. Thats been my personal experience, over the years.
The problem that has to be addressed is convincing people to care, and self improve.
Forcing a process on the unwilling is going to fail no matter what you do.
I don't agree that people who do care necessarily self-organise well, or that this self-organisation is compatible with other business goals.
Sometimes it works, and a group of engineers et al. do a good job of forming a coherent team with minimal process. Valve's maybe an example – I don't know. My experience is that this is the exception, rather than the common case.
A good process can provide tools that help good people who care communicate and deliver more effectively. They help us to keep track of how we're performing and analyse pain points, and to make sure that there are methods for dealing with common problems.
For example, if my team felt that daily stand-ups were an inconvenience that weren't helpful, then we wouldn't do them. That's the goal; there's no cast-iron set of rules, just a generally applicable set of principles to follow.
Some people are self motivated and other needs their slap in the morning (for some, that's a couple of times daily).
So where's my Half Life 3, dammit ;-)
Why is Valve a megacorporation vs. just a corporation?
Experience needed (read: got fucked before, now can avoid the treacherous waters and apply effort where and when meaningful).
Ability to deal with herding cats much needed as well.
Add relentless ability to move forward and not hold grudges.
Emphasis mine. You probably have more experience than I do, but largely I've seen it to be true that skilled (or, in the absence of skill, passionate) programmers tend to self-organize, as long as they all have the same goal (which hopefully is true for the majority of your team - if not, no methodology is going to help you).
All the companies I was at before this one that hid behind process had the usual problems with quality, scheduling, communication, etc. There's no process that's going to pull excellent work out of average people. It's just band-aids that keep people from taking real responsibility because they believe the Process will take care of things.
From my POV, a minimum amount of process is useful to keep things on the rails when the pressure goes on, so people don't end up doing things wrong. After all, unit tests, git and CI are 'process'...
"Any sufficiently complicated company w/o management contains an ad hoc, informally-specified, bug-ridden, slow implementation of management”--wycats
In a corporate environment, you tend to have some burn outs, some 'holier-than-thou', some paycheck collectors, etc. The biggest difference probably is that you can't choose who you work with. It seems Agile as I've seen is implemented around that, which ultimately discourages the behaviors described in the prior paragraph. It becomes one writing quickly to finish things for the manager, rather than writing things for your coworkers.
We even are using one of the new open source libraries that tests and enforces the process, it is called 'poison ivy' - it hurts, like all good software engineering processes do!
Edit: do the downvoters disagree? We don't think there will be books, courses, tools for this new methodology?
It's been mentioned on here a lot that if you have a skilled set of programmers on your team you probably don't need Agile. They are self-managed and tend to do the Right Thing (TM) on their own.
I have seen Agile work and when it does it's pretty fun. Things move along, issues are dealt with, and nobody panics when things get behind a little. Just move it to the next release.
I think the problem is that The Consultants sold Agile to enterprise dev shops and they really ate it up. I don't feel Agile works from the top-down but rather organically from the bottom-up. In fact I believe that is true for most programming methodologies like this.
And Scrum. Don't get me started on Scrum.