What kind of phase shift are you talking about? Humans cannot hear the phase of a single oscillation. Beats produced by two tones with shifted frequency are audible because the resulting amplitude is a function of time. But this is unrelated to timing and rhythm of music.
I think it's more like an "equivalent time sampling" effect, and the perceived 1ms latency is actually a perceived phase shift.