I've lived in the midwest in a small farming town where virtually everyone, mostly only excluding my family, was directly descended from people that moved there from the same small part of what is now the Czech Republic at around the same time. It was incredibly homogeneous culturally.
What you're saying is true.
But it's also the case that some those homogenous, snow white communities worry about jihad, even though there aren't any Muslims within a hundred mile radius of their house.
And it's especially true that when these small, homogenous communities diversify -- due to refugee placement or just economic growth and sprawl -- the latent racism becomes not-so-latent.
When non-whites did migrate to the area they did so to cities to find work and they weren't always met warmly or treated fairly, but those cities were a world away from the culture and environment of my youth.
Cultural conditions created an environment that was welcoming to white people and hostile to PoC.
Look at the history the history of the KKK in my home state of Indiana https://en.wikipedia.org/wiki/Indiana_Klan and tell me you don't see widespread organized racism in it's past.
We didn't see racial hatred growing up because our grandparents and their parents contributed to the conditions where there simply were not people of color. That and because subtle racial prejudices were more easily normalized by a distinct lack of larger communities of color.
I look at the emails I get from representatives in Indiana and I'm baffled by the issues they present: undocumented citizens, Islamic terrorism, etc... Communities in the midwest intentionally isolate themselves from people they perceive as "others". How is that anything other than racism?