I believe the issue people have with the term ex-pat is that it sounds like a fart-sniffing variation of "migrant" (or the derivatives "emigrant" or "immigrant").