You are forgetting the diversity itself is valuable. For lots and lots of reasons. One is that that the metrics that people who ignore diversity tend to use are easily gamed: GPA, fancy degree, etc., and may correlate more to privilege than capability. So by casting a wider net you put all the candidates in a more competitive setting, and hopefully force interviewers to figure out what they are actually looking for, not just things their gut correlates with good hire.

> You realize that you still are not selecting the most skilled candidate if you filter based on diversity at any level.

This depends on the size of your sample and the positions to fill. It is easier to find one good candidate from 100 individuals than 100 from 1000.