Any time I've pushed back on that racism, the person has redefined racism so that what they are doing isn't racism any more. Typically that involves the phrase "systemic racism" instead, as if the need to qualify the type of racism doesn't invalidate their point.

Unsurprisingly, racists aren't willing to actually discuss racism and instead do whatever they have to in order to justify it.

The more things change, the more they stay the same.