It's not just connection latency, it's decision making latency. As explained in the parent comment, all non-speaking mics are muted. If you want to interrupt, you have to make noise for long enough that it thinks you're talking now, before it will open up your mic. Otherwise it will think it's a cough or innocent tap on the desk.