I tend to switch to the language hinted at by the accent of the person I'm speaking to. And I only notice which language I speak when I can't find a word. The question then becomes whether I can substitute the word from another language familiar to the recipients.
In bilingual places you'll see people switching mid-sentence when they have a better expression in the other language for what they are about to say.
A friend who is learning Dutch once complained to me that Dutch speakers will switch to English the moment they hear you are struggling with the language. I suggested she should explicitly ask people to keep the conversation in Dutch.
In one instance, she did that, and the (native Dutch-speaking) person she was talking to agreed to her request, and then proceeded in English anyway! She asked that person multiple times, and they just kept speaking English without addressing her request. Baffling.