You are totally right. This is why I don’t even bother using polite form expect when I really need it: my interlocutor is glad I’m speaking more Japanese than he could speak English or my native language anyway.

I would also add that speaking a Japanese that is too good can backfire, as native have then more expectations that go further than language (behavior), especially for East-Asian people. Keeping illusion of semi-(in)competence can be useful.