I think a lot of people are opposed to sexism or racism precisely because they undermine meritocracy. If you take away fairness or meritocracy as a goal, you are left with chauvinism, supremacy, and tribalism masquerading as their opposites.