It depends a lot :).

For the tax system it was developped with lots of free software, and support purchased to French IT companies. It was hosted on a big (for the time) linux server farm.

Here are a few interviews and slides by the director of this project, unfortunately in French (didn't check automated translations, might work):

https://www.toolinux.com/article/jean-marie-lapeyre-copernic...

http://www.supelec-rennes.fr/sic/JOURNEES/05_10_20/lapeyre.p...

The free software community in France was very happy to see one major ministry doing this kind of projects at the time.

Can't speak for France, but in Ireland this system has been in place since it was done on paper. Most (all?) IT work for the government is contracted out to private companies.