I understand you have your personal suspicions. I used to have similar thoughts. But please understand a) this is a well-studied topic, and b) a major component of America's endemic racism is white minimization, deflection, and denial, something made possible by pervasive white ignorance.

As an example, you might read Loewen's "Sundown Towns". He covers the Nadir, the post-reconstruction period that includes events like the Tulsa Massacre. He demonstrates pretty convincingly that white-led ethnic cleansing was widespread across the US. Through a variety of historical records, including extensive use of census data, he demonstrates that in town after town, all across the country, white people violently drove out any black people and then used violence to keep them out. And then they quietly erased that history, despite keeping otherwise extensive historical records.

Was any of this covered in my high school or college education? No. That's even though it demonstrably took place in towns all around where I grew up. That a lot of suburban growth was driven by racism. That wealth distribution was driven by racism. That made it easy for me to think that racism was this thing that happened long ago or far away. When instead I grew up soaking in its structural effects.