I've had exactly the same experiences regarding mis-timed gaps among different groups; it can create an incredibly jarring experience, as well as a perception of rudeness by both sides.

In a more formal setting, you may have no better recourse than just learning to adapt, but with someone you're close to, it seems worth talking through explicitly.