It's more than just human ear sensitivity. Phase shifting changes the relationship between frequencies in a sound.

For example, phase shifting can change the amplitude of a sound. Analog hardware is especially sensitive to changes in amplitude so you might be introducing distortion just by shifting phase.