Yes, that’s my experience too. I have to choose during each call by opening the control centre and choosing the audio settings option for the ongoing call.

This does seem to let me enable voice isolation though, which seems to work very well.