Historically, this is quite wrong. By the late 19th to early 20th c., "white" identity in the U.S. had essentially become a way of providing all European immigrants with one common, shared identity thus obviating prejudice towards some European nationalities especially wrt. Central Europe (hence Germans, but also Slavs). To a significant extent, this is comparable to how self-identification as "white" is understood today, as well. There's nothing in such self-identification that implies prejudice against people of color.
Roman identity also fits within the ingroup-outgroup frameworm but not quite immutable. It involved such odd priorities by modern standards like retiring a "gaulic stereotype" class of gladiators after the conquest of Gaul because they were Roman now, even if many were slaves it became "politically incorrect".
https://www.reddit.com/r/AskHistorians/comments/758v1g/how_d...
> colonialism didn't really care about the colour of your skin
This is a step too far; if there's any home of white identity it's in the colonies that largely eradicated the natives, such as the US and Australia. Worldwide, people are racist against black people like Americans are. It's American tropes about black people that have traveled all over the world. Even the British, who had their own brutal African colonies, are racist towards black people through borrowed American language and imagery. Native British racism largely revolves around South Asia, just as native French (and Italian) racism is largely obsessed with Arabic-speaking North Africa.