As a Finn, I'd interprete this as trying to be helpful, and less about not tolerating accent. "Oh, they're struggling with Finnish, it's easier for all of us if I switch to English." In my opinion, this is one of the top reasons for difficulties in learning Finnish - it's difficult to get the Finns talk Finnish with you. One has to insist.