After living for many years in another country and being called an "immigrant", while hearing other foreigners being called "expats" I realized that people just create a mental rule for what each word means: coming from a rich country = expat, coming from a poor country = immigrant.

So an American permanently living in the UK will be considered an expat. A polish will be called an immigrant. Despite the actual definition of the word, "immigrant" has a negative nuance artificially attached to it so people use it to this effect.

In Europe I noticed that skin color makes less of a difference than "source country". Many British people will still treat African-American individuals as expats, and an Albanian as an immigrant, once they find out where they came from.