Many physical attributes are absolutely relevant for a given context.
What we should do, is getting rid of irrelevant and abhorrent discrimination, such as racial selection.
Society made progress only because people fought against that. They were all accused of fighting against nature, but over time, more people realized those old standards weren't as "natural" as they were once thought to be.
Do you imagine we've reached the pinnacle of human social evolution and tolerance? Perhaps we still have similar lessons to learn, and people of the future will consider our behavior as abhorrent as we consider the 1800s.
The only progress that could push society past that, is genetic engineering. Everyone on equal footing or so different, that sport as we know it ceases to exist.
But that's no reason not to eliminate discrimination when we can.
But if a group is excluding people based on innate physical characteristics, maybe they should rethink that.