(I worked to set up a UK public sector website back in 2004/05. We cut our costs by 90% - genuinely, 90% - by bringing all development in-house rather than using private sector contractors. I do not believe your assertions to be consistently true in the UK.)
In the private sector, shareholders take money out of the system in the form of profit. Something that costs £X in labour and consumables must be charged at £X+Y% in order to make that happen and still give money to investors. If they can do it cheaper, they still charge £X+Y%, and give the surplus to the shareholders.
In the public sector, there is no shareholder. If it costs £X in labour and consumables, then the cost to the public purse is £X. If they manage to do it cheaper, then that money remains in the system and can be used to either improve the offering, or subsidise another project.
Public sector workers are also often paid considerably less than their private sector peers, and departments are often highly restricted in their ability to pay market rates for specialist employees.
Although the total benefits package for public sector workers has historically been equal to or better than their private sector peers, this has typically been due to Final Salary pensions, which have been mostly phased out in favour of Career Average or even Defined Contribution pensions, all without a corresponding increase in salary.
The cost to build something in the private sector is almost always cheaper than asking government employees to do it. The public sector "didn't build it" like you said, but if they had, it would have been cheaper. This is almost always the case.