Your comment made me realise that I actually never switch with the pods button (rather on the display of my phone) because of the loud noise. I don't get why they have not fixed this after years of complains from users.

Doesn't it still make the noise when you use the onscreen switch?

Either way it's horrible, all of these noises seem to be at a fixed or separate volume level than the media volume (which actually is common for apple). Siri used to have a separate volume than media (still might?), and then you also have ringtone volume, and sound in settings barely even hints that all these separate volumes exist.