Everything you say is essentially also valid for immigrants (perhaps now citizens) who haven't lost their accent.

While I agree this is not ideal, it's also built into human nature to use any/all hints to help generalize/categorize (likely because it determined survival in the past).