From my experience, macOS just seems to be really conservative with avoiding drop-outs in the audio. Are you using the speakers for music, or more general use? I've found it tends to value quality over responsiveness with music in particular.