This happens most often on calls, since battery life is much worse (60%?) when speaker and mic are being used. That’s why I’m left scrambling to get the other earbud in — someone is talking whether I can hear them or not.