There’s apps (it’s also pretty easy to write one yourself like I did) that monitor when the input source is changed and forces it back immediately. Yes, wish that feature was built into the OS, but I also don’t grant almost any app microphone access.