I just tried this and you're right, complete random phase across the the whole frequency band does not noticeably change things. Funny that we learned that differently. Thanks I learnt something about audio today :).
afaik phase only really matters in audio for speaker enclosure design, placement, and likewise microphones.
However, because its virtually unknown to audio folks theres perceptible nodes everywhere, if you can hear them.
The same is not true for radio. There, corrupting the phase relationships corrupts the data (for many systems).
Phase is relative, but our ears don't care about relative phase either (at least as long as you don't stick nonlinear filters after, then it starts mattering, but usually in audio things are fairly decorrelated anyway so it only matters in quite specific cases).
Here is an example: https://twitter.com/marcan42/status/1282685645731672064
Demo: https://twitter.com/zwegner/status/1282859889447116809 (interestingly, you can hear the change in Twitter's low quality encode, but it goes away at higher qualities, so it seems their crappy AAC encoder does care about relative phase :-))
It's counterintuitive how little our ears care about phase across frequency bands. This is not true for other kinds of signals.