The problem with not having affirmative action is that disadvantaged groups, even if on a level playing field on paper, have had their wealth actively destroyed while the advantaged groups have accumulated lots of it. The civil rights movement made people equal on paper, but nonwhites still had to deal with the shit hand they were dealt for generations. (And even legally advantaged groups have subgroups that have not done well, like Appalachian whites, but even so they were eligible for things like GI Bill benefits that nonwhites did not.)

Affirmative action is a clumsy policy to fix the legacy of such issues, but more targeted ones like substantially increasing education and social funding for disadvantaged groups, or reparations, are politically beyond the pale.