I have the same thing here in Canada where people switch to French while I want the conversation to stay in English. I just say "Désolé je ne parle pas francais"(sorry I don't speak french) practiced to sound like broken french.