> If total latency ~ 500 ms, phone-to-tower is maybe 50ms of that in 4G. Even if it's down to ~1ms, total latency is only down 10%.

How did you come up with 500ms for comparison? A fast-paced videogame with that sort of latency would be unplayable. 100ms is still considered bad and 50ms is considered acceptable.