That was true a couple of years ago, but healthcare.gov was a wakeup call, and everybody from the president on down takes reforming government IT seriously.

I was part of the healthcare.gov "tech surge" and I can tell you firsthand that the government is willing to change. In the last year, the healthcare.gov UI was streamlined from 76 to 16 screens for the average user. Large parts of the site now run on AWS and more is moving. The most problematic individual component, the login system, was rewritten from scratch with an API-compatible, drop-in replacement which now serves 100% of traffic. And this has been done within the same agency responsible for the original site. It is not a story of "smart hackers" parachuting in and wresting control away from government bureaucrats, but rather partnering with the agencies and creating reform from within.

USDS itself, although started by the White House, now has bipartisan support and a real budget to grow. Although you are right that there are powerful entrenched interests in the beltway, the political will to change is there, and now the challenge is to recruit people like Andrew to seize the opportunity.

So, you worked for a short while on an extremely high-profile project with the personal daily attention of the president. Yes, that does tend to smooth things over - but don't mistake that for a fundamental change in government priorities. There are 2.6 million federal employees, you don't re-jig politics and incentive structures that has been entrenched over decades overnight with a single successful project.

Sure, showing that it can be done is always a good first step, and I'm not saying that anyone actively wants to fail - but only just barely. More often than not it feels that a failure you can blame on someone else is better than a mediocre success (good luck running an election campaign on having reduced the number of screens in a system or migrating it to AWS).

Why don't you ask the people who work at USDS if they're having trouble getting interest from the agencies? That would be a way to test your hypothesis.

But my question would be this: Are your political sponsors (a number of rungs up) invested in your work? Are they willing to take flak for your team (ie. take personal career risk) when (yes, when) things go wrong and provide the cover to help you turn things around, or are they more likely to throw the cocky Google-kids under the bus and declare "experiment failed" the second things turn a bit sour, and return to business as usual?

Now, answer the same question again, but 18 months from now, on the hypothetical that the next president isn't going to have this much on the agenda. What happens when heading the USDS doesn't mean briefing the president in the oval, but rather briefing a junior staffer to an undersecretary of the treasury?

Getting positive answers to all those is what real change looks like.

Maybe if the President has his personal eye on a project something could get done. But the President isn't going to put his personal eye on the USPTO back-end for example. Or the FAA exemptions submission page. Or any of a thousand other points of contact between the public and government.

I'm curious what, if any, security obstacles you faced with this, especially considering HIPAA and all that.

That environment has its own security accreditation

http://aws.amazon.com/govcloud-us/.

Our security audit was apparently the first one in years to pass with no "high findings" (high-priority security issues that must be fixed before the system is approved to go live).

The main barrier we ran into was that although AWS was already FedRAMP-certified, it had not gone through CMS's own internal security review process, which requires hundreds of pages of additional documentation and an audit of the code, team, and penetration testing of the live system. At the end of that process, you get an Authority to Operate, or ATO, and that's what every site in the government legally must have in order to launch.

You can get a sense of how much paperwork is involved here: http://www.cms.gov/Research-Statistics-Data-and-Systems/CMS-...

Navigating this ATO process was the single largest roadblock in developing the various parts of healthcare.gov 2.0. It meant that what we expected to take two months ended up taking more like eight.

The good news is that once we had gotten an ATO for AWS [1], any project within CMS--not just healthcare.gov--could start using AWS. [2] And within weeks of the ATO, we started hearing about other groups building their new projects on AWS, including Accenture, who's now the primary healthcare.gov contractor.

And that also means that if a startup wants to work with CMS, rather than the typical set of DC contractors, now they can use AWS too and have one fewer barrier keeping them out of the system.

[1] as a small technical point, we split the ATO into "infrastructure" and "application" portions so that the infrastructure (AWS) portion could be reused.

[2] If you're wondering why it matters that the government can use AWS, it's because most data centers that cater to the government are really terrible. In the initial data center used for healthcare.gov, the way you provisioned a new server was to send a word document to a sales representative, listing the unix packages you want installed, and then a few weeks later they would give you a virtual machine which may or may not have what you asked for. You can't be agile at all in that type of environment, and you certainly can't do DevOps well because you can't script anything.

I recently finished a project whereby it took over a year, from kickoff to handoff to deploy two blade servers.

Government IT is often extraordinarily hamstrung in the types of solutions it can deploy due to mandatory requirements to use certain security software, or employ specific policies. [1] While those policies come with a framework for adapting them to the needs of an environment, most don't understand them, and so you're often stuck with a computing environment that any DevOps engineer would find to be practically broken. There's an enormous amount of manual steps and needless shuffling between systems to accomplish the most rudimentary of tasks.

This enormous inefficiency is accepted as being inevitable. Because simple tasks take so long, and making them more efficient would require enormous effort (such getting exceptions to policies approved), the issue is often solved by adding more people. Contractors make their money through staffing, so if you can show that you can get more done with more people, then it can be a good sell. Costs, however, go up. This is where your overruns come from.

(Regarding ATOs, that sounds like the "Type" vs. "Site" accreditations I'm familiar with.)

The reason you have the same companies has very little to do with "insider" status and much more to do with the fact that these companies have made it their business to be able to operate in the federal environment.

Bureaucracy, far from being a conscious, malevolent tool, is simply the accumulated barriers and obstacles erected over time for every instance there was an anomaly or problem. Accountability was demanded, and new guidelines put in place. These stack up over time, long after the original incidence has ceased to be a major issue. Unlike the private sector where companies and organizations can simply be wiped out and start with a clean slate, government doesn't (and for various reasons, can't) do that.

There's still money to be made however, for those who are familar with the process. Simply being eligable to bid for government work means getting approved on a contract vehicle, complying with various laws on accountability and record keeping, and ticking off various checkboxes for equal representation of genders and minority groups. These regulations are removed and unfeeling of special circumstances or facts on the ground; they are the accumulated "good intentions" of laws passed that now stack up like strata.

People who know this system well are intimidated by all of this, which is why you see many government and defense retirees move on to the private sector to steer these small companies into this kind of work. They know how it works, and just knowing that is a skillset in and of itself.

It's not evil. And there's far less cronyism than one might suspect. (I'm not claiming for a moment that it doesn't happen, but somewhat surprisingly, it's minimal.) Improvements are not consciously forbidden, and making things better is not discouraged. But like an enormous traffic jam, no single person has room to do much of anything. Coordinating a number of people to try and move in a better direction can improve things a little, until they are stopped by another group who has no room to move.

It's a mistake to think of government IT as insidious. It's a problem we have built over time that we have created ourselves. To fix it will require enormous bravery to do away with volumes of regulations that, by themselves, probably did some good. And it will require tolerance of practices done in private that would never have seen the light of day, but in government, has the spotlight of oversight shining on. It will be a challenge to solve.

Yes, of course the companies that get government contracts need to be experts at getting government contracts -- the problem is that they do not need to be experts at what they actually _do_, what they're getting paid for. There's in fact very little room left to be experts at what they do, after all the expertise they need to develop (and resources spent on same) in getting and keeping contracts.

If you're a company and your motive is making a profit, then of course there's plenty of money to be made, exactly.

The system you describe sounds pretty insidious to me. "Insidious. adj. proceeding in a gradual, subtle way, but with harmful effects."

The harmful effects are exactly the privileging of expertise at government contracting over expertise at doing a good job. That's harmful only if you prioritize doing a good job, if you prioritize making a profit I suppose it's just fine.

> But like an enormous traffic jam, no single person has room to do much of anything. Coordinating a number of people to try and move in a better direction can improve things a little, until they are stopped by another group who has no room to move.

Yes, quite, that's quite insidious, exactly. It has some unique features to government, but also some commonalities with any large bureaucracy.

You're right in that expertise in getting contracts often has a tendency to displace expertise in IT. This has been a persistent problem in hiring. Contractors need to fill billets that come with mandatory requirements dictated by the government. Skilled individuals are often bypassed because they lack a certain certification (often unrelated to their work) or a certain parameter. In defense, security clearances can be an enormous barrier by limiting only those who possess a clearance as eligible. (There's an "insidious" effect whereby you must be on or joining a project that requires a clearance to get a clearance, and you cannot join a cleared project without a clearance.) Pay is often much lower than compared to the private sector when compared to the industry as a whole, but since the layout of the federal government is separate than what drives the private IT industry, it creates opportunities for many where a grassroots IT industry wouldn't otherwise exist.

There are good people who enter the environment and make positive change, but they are few. There are many more who work at a perfectly capable level and do the best they can, but would be considered underqualified for their role. This is an area where the Digital Service seems to me trying to make a difference, as government IT rarely attracts the IT elite.

That is a great metaphor. Anybody who's considering joining USDS, I think ckozlowski's comment gets the situation exactly right. You are unblocking a traffic jam.

I know -- very condescending, isn't it? Gee, it's a good thing these really smart folks are showing up. What would we have done without them? A broken system must obviously be full of incompetent people.

I wish these guys the best, but this is 90% bullshit and 10% reality.

Just to cite one, small (?) problem: in commercial IT, if you don't like the work somebody is producing? You fire them. In government IT, no such thing exists. Managers spend years shuffling around poor-performing employees in a desperate attempt just to keep the lights on.

I could go on, but it would all just wash over folks. This is a movement, dammit! We all feel so very excited! This time is different!

So I will help do whatever I can -- blog, comment, debug, whatever. But I will not become suddenly blind to the realities of the system that we are attempting to change. That doesn't do anybody any good. That ain't cynicism, that's reality. Hope is not a strategy, and you can't fix a system you can't honestly describe. (or understand how it got that way)

My own perspective: There are amazingly smart people working in government, and even working for many of the contractors commonly cited as examples of The Problem. Many of these individuals are not effective not because they're incompetent (and therefore need to be told what to do), but because they are not empowered, or they lack confidence or a team of people they can use to propel good solutions forward. Bureaucracy and internal politics are rampant. One way to look at what we're trying to do here is to act as a nucleus of tech expertise inside these agencies, to attract and empower those individuals to do the right thing while providing air cover from the White House when they're prevented from doing so.

That being said, there is a vanishingly small number of actual engineers working for the government. Normally this role belongs to the contracting companies. The incentives here frequently encourage overly large, excessively complex designs, and implementations that maximize the number of humans rather than good engineering practices. By improving technical literacy within government we improve the government's ability to identify bad contracts and methods to prevent these situations from happening in the future.

We are not blind to the fact that there have been other attempts before us. The key differences I see are (a) support from the top and (b) funding.

Best of luck!

If a lot of these things were done in-house in the government itself, things would be a lot more efficient.