I'd argue that it's not the jobs of universities or employers to try to "fudge the difference" between actual outcomes and what they believe outcomes should be. By that I mean, just because not everybody has equal access to education doesn't mean employers and universities should be lowering their hiring standards (like it or not, this is what affirmative action necessitates) for people belonging to groups that on average come from disadvantaged backgrounds. Not only is it bad for business or the academic reputation of the university, it can also do more harm than good by placing people from disadvantaged groups in environments they are not prepared for, harming their growth and unnecessarily exposing them to discouraging failure.

So by that I mean, why can't we just evaluate individuals based on their own merits? If you want to stop inequality, fight to stop inequality of opportunity. Stopping inequality of outcomes makes little sense.