All the most popular links on gov.uk take me to a very nice landing page with an overview, but if I actually want to do something they all link out to the existing web apps. Some of which I've used and are not terrible, but not likely to win many design awards either.
It's great that the UK government is no longer consistently failing at the simple task of putting information online, but anything past the 20th century web still seems beyond them. Hopefully it's the next step, I don't really envy the people that need to make that cultural change happen. I do however feel it's more likely to happen sooner if we don't pretend that it'll just happen magically by itself.
I did spot a mock-up of a form for reporting lost Passports, which when I googled for it was a nice 3 question form which after processing your answers used that information to give you a link to the correct PDF form to fill in and post (ok, to be fair it gave some helpful extra contextual info like the relevant countries embassy info, but to be unfair if you ask it about an Adult passport, then want to know about a child passport--because I'd imagine losing your whole families passports at once is relatively common--it forces you to answer the remaining 3/4 questions again).
So, some little steps in the right direction as far as web apps for interacting with government are concerned.
The old website only persists because a few remaining functions have not been transitioned yet. It's rubbish, but it's actively being replaced. Criticising it is pointless.
It's interesting that you refer to alpha.gov.uk. The gov.uk site left the alpha stage and officially launched in October 2012, and as much as government IT projects are usually a disaster, you can't deny gov.uk is a major step in the right direction.