“Immediately” is not the word. As the phone is answered, it’s trying to automatically switch. Pressing that key while it’s in the process of auto switch just confuses it, and typically stops the switch process altogether. Then you have to do it again.

Fwiw, this seemed to work smoothly two years ago. And then… updates. AirPod firmware updates. iPhone updates. macOS updates. Nothing works the same as it did 2 years ago for maybe … 2 months or so. And there’s no going back. :(