> A lot of this is because railway signalling operates on a brick-wall principal

That applies to old style -though still in common use- signalling. CBTC (every railway should use, but very few uses it) has similar, or even tighter, margins to road signalling.

No, the gains from CBTC come from two factors: one, it being a moving block system, rather than an absolute block (really you can think of a moving block system as an absolute block one where the length of the block approaches zero), so spacing matches the braking distance required; secondly, as with many modern in-cab systems, it is down to the individual trains to compute their braking curve, rather than the length of the blocks being dictated by the worse-case braking performance of any stock on the line.

I'm also unaware of any freight or mixed traffic application of CBTC, which makes it a stretch to say every railway should, though plenty of proven in-cab systems provide many of the same benefits (and you can decrease block-length substantially to get much of the way there).

But that not just some weird quirk railroad engineering clings to because Musk has not disrupted them yet, it's a consequence of braking performance.

Drivers feel safe to attempt brake matching (they fail often enough) because road code assumes that you never go fast enough to make stopping distance exceed visual range. Even if that rule is routinely broken the brake-match distance stays comfortably within visual range (stoplight waves travel upstream). In rail, everything happening within visual range is basically too late to even bother and this is entirely a consequence of braking performance.

> In rail, everything happening within visual range is basically too late to even bother

Not really. EMU passenger trains, such as subway trains, do have good acceleration times--good enough that it's limited by passenger comfort, not by physical hardware. This limit is about 1 m/s², with emergency brake conditions reaching 3 m/s² (note that the latter does imply several passengers are going to be nursing injuries--there's no seat belts after all). That's roughly comparable to typical passenger vehicles.

Freight trains have much longer braking distances, but that's a factor of 10,000 tons moving at 50mph has an insane momentum combined with relatively few axles being able to contribute to stopping force.

The main reason you need large distances between trains: switches. To control where a train goes requires moving a physical piece of infrastructure at the switch. You can't move the switch until the previous train clears it, and you don't want to let the subsequent train reserve a path over it until it switches into a new position--if the switch gets stuck in the middle, the train derails instead (or worse). The "brick-wall" principal follows from this situation.