And if the decision isn't based on race, but on things that happen to correlate with it? Whether your father was present, for instance - since that's a major predictive factor for all sorts of life outcomes even when controlling for race. Should that not be used just because certain groups are more likely to come from single-parent households than others?

Because in the end, the choices are "accept that there will be a correlation between the outputs and race", "use no system that produces estimates or imperfect outputs", or "explicitly discriminate based on race to remove the race-result correlation". That's it.