A lot of the systemic issues that have caused people of certain races to be poorer and less educated have either a historical or direct basis in race. And then if you look then at the poverty rates by race, it's clear that certain policies (esp. with regards to educational funding) often have a far worse impact on members of a certain race than others.

It's not hard to make a connection between, say, an actually racist issue like redlining and how it affected the development of neighborhoods, and the subsequent lack of local funding for educational programs in the affected neighborhoods.

Sure - but none of this implies that changes are required to math education itself, or that there is anything racist about how math is taught. It just means missing resources should be provided.