As i understand it, there's a rule that requires a member of staff on any train that's properly underground. Ostensibly for safety in emergencies, but possibly because the unions fancied the jobs.

On current tube and big railway trains, that member of staff is the driver. On the DLR, it's the train captain. On an automated Crossrail train, it could be a guard (it's buses that have conductors!).