Sound travels around 1.7 meters in 5ms. Acoustic musicians can play fine while being apart more of that distance, although there is a tendency to slow down, unless consciously keeping the tempo up (everyone individually feel that they are rushing a little bit, but that's how they just keep the pace).

There is a huge difference between consistent latency and jitter, but even then I doubt that 1ms latency + 1ms jitter would be very noticeable.