Neutral activity itself definitely takes phase into account somehow, since you can hear the phase difference of two same-frequency tones being played independently into each ear, without any kind of sufficiently strong mechanical connection that would explain it.