One of the major problems (at least in the US) is that the process is heavyweight and specifically tuned towards "lowest conforming bid". You end up getting subpar work and long delays, but the government is absolutely unwilling to restart the process in almost all cases because the cost is just that large (and we picked this one because it was going to be the lowest cost!). In some ways, the government has very little control over the outcome.

We've seen through things like 18f and usds that the government can adequately and effectively manage software projects itself and produce amazing results. Giving the US government more agency to self-fund ongoing development and procure solutions that are technically superior as opposed to "lowest conforming bid" is what is really needed IMO.

The "lowest bid" thing is actually now starting to change. Government buyers are starting to get trained--and there is an accompanying culture shift--on the idea of best value, which is a combination of price, timeliness, ability to deliver, and whatever other factors actually contribute to money not being wasted.

It's stupid that for so long it's been coded into law that you need to spend as little as possible, because that's part of what got us here; the buyers in a lot of cases can't go with a better, more expensive option.

Hopefully the "best value" culture shift in government buying will go along with legal changes to let them act on that.

Still the problem is that when Lockheed Martin rolls in with a $350million bid, Boeing comes in with a $365 million bid, and Joe Schmoe's Web Shop shows up with a $450k bid, they dismiss Joe Schmoe as trivially worthless despite the fact he's got low overhead, had built systems exactly like what they need, and could do it quickly. LM and Boeing, on the other hand, are bumbling fools, giant towering piles of middle managers and upper managers and the worst examples of bureaucracy you'll ever face. The kind of people who if faced with a situation where the company has already been paid but the work location is going to get shut down for a day they will try to trick workers into taking their own vacation time to cover it, simply conveniently not mentioning that the time will be paid anyway. Sure you could just be decent, but if there's a chance to gouge, GOUGE AWAY. But they look 'respectable'. I've seen a billion-dollar contract where they found an off-the-shelf product that did exactly what they wanted... so they bought it, gutted it, and rebuilt it more terrible than ever in order to justify the expense. I've been on one side saying 'you guys will mess it up, just let us develop the new version' and been told 'you can do it... but you still have to pay US for writing it. Because that's part of the contract we won and we'll sue you to death otherwise.'

Part of the problem is random bureaucrats have no way of knowing if Joe Schmoe's Web Shop can actually do the job. Some of it has to also be covering your ass - if there are serious flaws in the system, your ass is covered if you spent $350 million - not so much if you spent $450k.

I have no idea how the process goes, but it seems like some sort of outside consultation on picking appropriate bids might help. Maybe even something like the congressional budget office, but for software systems. That way expertise can be concentrated and they can focus on their core competency, rather than requiring every agency to develop people who can properly analyze bids for software systems.

Another major problem in consulting projects is accountability. There is no central repository of data on major consulting firm's success rates, and there is no way for most administrators to objectively determine the risk associated with a given consulting project. However, there is significant risk associated with the failure of an in-house initiative from the perspective of government employees, just as there is in the corporate world: it is easy and low-risk to blame contractors for failure, but if you are the one managing development there is no one to blame but you.

If I may commit sacrilege here, I think the problem is more market based than government based. The problem in many cases is that technically incompetent people are in power who view software as a commodity and choose the lowest priced version, no matter the later problems. I'm not even sure what the solution to this is.