given my experience with thorough, to the letter, under supervision of no-expenses-spared-top-notch experts implementation of Agile/Lean/Scrum - 2 years non-stop across pretty much the whole company, only very-super-important-critical-projects got exception - the government would love it - a lot of planing, reporting, meetings and other crap boggling down everything with actual product development slowing down to the crawl as a result.
Btw, the original healthcare.gov - the one that failed and required that emergency super-team fixing - was developed using Scrum, and as result it had all the characteristics of a Scrum project, chief among them - bad, to the point of straight failure, integration between components.
In my experience there are two factors required for a successful project:
a) Developer autonomy sufficient to modify core design/implementation if necessary--and developers experienced and talented enough to recognize when this is necessary.
b) Developer ownership of the project. By that I mean minimal power difference between "stakeholders"/"business owners" and the developers actually building the product. You don't want developers to feel like consultants working on someone else's project because they won't care when bad decisions are made and they will naturally do the least amount of work possible to satisfy each Scrum ticket.
However, the large amount of bathwater doesn't mean there isn't a baby there. And the places I see doing best at making the necessary culture shift are the ones with a sense of mission. So I have hope.
And honestly, even if this merely means a transition away from pure Waterfall to Mini-Waterfall, I think that will be a step forward. In the commercial world, the rise of the Internet has made 3-year release cycles untenable for most software; people now just scoff. So if we can merely get governments to emulate business in the shift from titanic failures to medium-sized failures, as a citizen I think that's a win. But still, from what I've seen, I'm hoping for more here.
Applying generic systems analysis (where it doesn't matter whether it is protein building, network protocol flow or software engineering process) one can easily see that Scrum is a highly synchronous low-latency process (that is the "baby" that management loves) which comes at the cost of dramatically decreased bandwidth. In addition, changing a team's process inside, Scrum indirectly completely changes how the team interacts with up- and downstream teams/components - in particular on practice eliminating any chance for early integration and for any iterations of the integrations.
That became obvious to me during the first day at the first Scrum course, especially after i tried to ask those questions to the teaching Scrum expert from the consulting company and received blank face with some blabbing in response. Some time later, different company with different consultants - that first hand 2 year experience i described in the previous post - the Scrum drawbacks on practice happened to be even worse than i thought they may in theory. We actually have great time right now at our BigCo. as after the Scrum fiasco, and quietly dropping the Scrum company-wide as the result, the management still seems to be kind of "in shock", and they haven't brought up any new fad in the process improvement so far, so we're just doing typical old-fashioned per-feature waterfall, and it just works as usual - which is huge achievement compare to Scrum :)
I think that's especially true for dramatic company-wide, top-down Agile adoptions, which I have never heard of working. I think people doing that cannot possibly understand Agile philosophy, which promotes local control and continuous improvement of top-down control and giant leaps.
My positive experience is mostly with smaller shops, but I've definitely seen many places where companies have grown up releasing early and often and have no problem working in what I'd consider good Agile style. Wealthfront, Etsy, and Spotify are all explicitly and openly like that. But I think there are plenty of other places that aren't explicitly Agile but ended up that way as a natural evolution.
For example, when I visited YouTube to study their process, there were a great number of developers, designers, and product managers all working in relatively independent small teams. They'd release at least weekly but sometimes more often. There was no overarching plan; each team had particular goals to pursue, and coordinated with other teams as needed. They never used Agile jargon and didn't think of themselves as Agile, but I think they were acting in ways that exactly matched the spirit.
EDIT: changed this sentence: "Much was organic and some top-down mandated" to reflect what I think was mostly due to popular upswell, though there were key executive decisions. The O'Reilly network has a lot of influential folks in DC too.
Imagine politicians with a stake in the project's failure hearing that: they will go full Benghazi on it.
You don't want to be seen having failures in your project (even small, agile failures that are for the good)... but you don't care if the project succeeds.
If you can plan out a waterfall project that won't complete until you're "out of office", then there is no negative consequences for you... even if you KNOW it will fail once you're gone, that's the next guy's problem.
Sad, but true.
Back in 2004 or so, I heard of a major company that only updated their website once a year. Even then that was considered ridiculous. Consumer expectations have shifted massively; plenty of places now release every week or two.
I have hope that the government culture could shift similarly, so that software projects that don't deliver anything for long periods are seen as troubled no matter how many Powerpoint decks and GANTT charts they produce in the meantime.
And I think the way to change that culture is by having enough projects that deliver early and often, so that people say, "Well, those guys are shipping monthly. What's wrong with your team?" So instead of keeping people from being blame-y, we get them to be blame-y about long project cycles.
I think that's definitely happened a fair bit in industry, so there's some chance it will happen in government, too.
If it's either a failed public project, or spending millions of dollars (that can be spun as creating jobs), what do you think they're going to take?
The challenge that USDS will need to overcome is one of culture and regulation bureaucracy, not one of throwing a unicorn cavalry at various IT dragons. You go pure "change from within", you'll fail. You go pure newbie/private sector, you'll fail. You need to find fresh and innovative eyes and pair them up with existing Fed ITers that haven't yet been jaded into submission.
Like you, I'm excited to see how it turns out. They need to succeed. If they fail, it will be a long-term failure, and that will do no one any good.
The question is: Can he convince others in government that, as another post on the HN front page says right now, we should replace middle management with APIs?
[0] https://github.com/18F/api-standards
[1] https://github.com/18F/jekyll_pages_api
[2] https://github.com/18F/API-All-the-X and http://18f.github.io/API-All-the-X/
[3] https://github.com/18F/api.data.gov
[4] https://github.com/18F/fbopen
[5] https://github.com/18f/?query=api etc, etc
But then I don't think that should be a goal. (Good) middle management is precisely what Washington needs more of.
[1] The author would be well advised reading Peopleware, and understanding that middle management is often (usually?) an exception handling layer that handles the boundary between systems.
One of the most interesting bits to me was something said by Todd Park, a former CTO of the US and still a White House advisor. I was, frankly, suspicious that Washington's culture could really accept a lot of the Agile and Lean notions that are commonplace in the high-tech world. But he pointed out, correctly, that waterfall projects are enormously risky. A big reason Lean Startup advocates place such a strong emphasis on frequent small releases is that it helps us reduce risk by finding and fixing problems early. He pointed out that if bureaucrats really want to play it safe, using agile, iterative approaches is exactly what they'll have to do.
I don't know enough about Washington to say if this really will work, but improving government efficiency through smart use of tech strikes me as exactly the kind of thing people from all parties can get behind. I'm excited to see how it turns out.