I think it's because those with political influence who espouse xenophobic views also tend to dislike the poor. "Taxpayer" as an identity has connotations of "only the landed gentry should have a voice in society's management", whereas "citizen" connotes equal rights and a voice in society regardless of income.