There's also a difference in volume and freq response due to the head being in the way and also the shape of the ear that helps you determine the direction.

I don't think we could tell direction based on the distance and less than 1ms difference alone.