Your argument is mostly "whataboutism" - you point out that terrible things have happened to groups of people in the past and those things have inter-generational effects. Of course it's true.

That being said, of course it's a matter of degree, but at that point you're just arguing semantics. Driving your car at 25mph vs 125mph is only a difference of degree, and yet one is totally permissible and the other is incredibly dangerous and would land you with a big fine. The key point here is that speeding becomes increasingly more dangerous as you drive faster, to the point that exceeding speed limits by a given amount becomes reckless driving because you're much more likely to cause injury and/or death to yourself and others. In the same way, racist systems create a compounding disadvantage that is almost impossible to rectify simply by changing the laws to make them "colorblind" in the modern day.