They are effectively building an express system as seperate lines. E.g. the new Elizabeth line is effectively an express Central line.

Isn't the problem with automation on the tube less the technology but more the unions ?

No, it's the technology or perhaps the companies supplying it (unions are only ever a problem in the US - or maybe they're never a problem and there's only a propaganda system against them in the US). London Underground are currently in the process of literally the third attempt to upgrade the signalling on the subsurface lines (i.e. including the District) - the previous attempt by Bombardier failed outright (as London Underground were aware it would from the early days, but they were politically obliged to wait until the company admitted as much), and the one before that (a similar story with New Labour ideology-driven PPP) was also abandoned.

Full automation—without drivers—would typically require platform edge doors, which are expensive and troublesome to introduce without disrupting service, and very difficult on some of the curved platforms.

The benefit for the cost involved really just isn't there; while drivers aren't cheap they aren't impossibly expensive in comparison with the average number of passengers per train.

The unions don't help but they reflect largely the interest of their members. Who doesn't want constantly improving pay and conditions and no possibility of redundancies? In effect you have to bribe the train union / its members to land automation that they perceive as potentially weakening their future bargaining position. They don't care about notional safety -- or at least no more than most individuals do in practice -- they care about their personal outcomes. One common bribe for drivers is time set aside for "education" that can be done at home and isn't assessed. Say you give drivers 30 minutes per week for this "education", well, now you've got 0.5 hours per driver of driving needed, you can't hire the necessary drivers quickly enough to cover that, so, you pay overtime to cover it. Lo and behold every driver is keen to do the extra overtime. Now your workers are doing the same job for more money, but when asked about it their union can say it's not that you paid more money, it's because you agreed they needed time for appropriate "education" which is quite different and will contribute to improved safety.

My favourite example of how unionised workers whose union invariably presents their preferences as "safety" care little about actual safety was signallers. Historically some mainline rail signallers (same unions) in the UK worked 12 hour shifts at outlying boxes. So you maybe do 3 x 12 hour shifts = 36 hours that's a week's work or take an extra as overtime for 48 hours. Roster maybe a dozen people to work a box and it remains open 24/7. Annoying scientists said, wait a minute, humans don't remain useful and attentive workers for 12 straight hours, especially at night which explains these incidents where a signaller makes a grave error at like 0400 after 10 hours at work. So the safety regulator wants to limit shifts to 8 hours like for air traffic. The unions are apoplectic because if you're a member working a mixture of 3x12 and 4x12 and now they want you to work 5x8 that's up to two days per week extra. Suddenly railway safety vanishes as a concern...

It is possible to make gradual progress. When I was born there were still guards on tube trains. Why? Well there had always been guards on tube trains. Today of course London Underground does not have guards. It just took a long time to make it happen, with I believe the Northern Line being last to stop having guards.

It is mainly due to an enormous expense and wanting to keep everything running during upgrades.

If you are already costing the taxpayer X million per year and then you want another 400M for an upgrade, are you likely to get it?