or TL;DR: bias is mostly a learned behaviour, with probably some inherent subconscious bias from our evolutionary roots.
The majority of people where at what times?
Does it actually matter why people hate, only that they do?
Your argument seems to be that all racism etc is systemic and structural, and therefore there's no individual responsibility?
Do you expect me to be an expert on the moral history of every group on earth? I'm not going to specify in exact detail, but my point is that I think people naturally default to ambivalence and have to learn (or be taught) to mistreat specific groups of people.
> Does it actually matter why people hate, only that they do?
Yes, it matters if you want to do anything about it.
> Your argument seems to be that all racism etc is systemic and structural, and therefore there's no individual responsibility?
Yep, pretty much. Or at least, most racism. Individuals are responsible for forming the opinions but to make any real difference you need to take away the reason they formed the opinion.
If you want to reduce oppression at scale, you need to address the systemic issue. Do you think a racist hates some set of people arbitrarily? Or do you think he might have a (obviously invalid) reason to hate them, such as feeling they are a threat, are inferior, can be exploited, are ridiculous and so on? Dig down into those reasons and I believe you will find a systemic root most of the time. Obviously that person shouldn't hold those views, but how are you going to address the problem? Attempt to force everyone with a bad opinion to change that opinion? Or do you address the reason they acquired the perception in the first place, meaning that the next generation has less reason to discriminate? I think the latter is more viable.
> exclusionary behaviour are the exception and most people abide by the golden rule
So the question for this has to be: when do you think the date was at which "most people" started to do this? Was it a mass conversion at the enactment of the US Civil Rights Act, or a much slower battle?