...it's math. If you hire from 60% instead of 100%, you're intentionally eliminating potential candidates. Whether you're talking about immigration policy or diversity policy. It's just math.
On fraud, I hear just as much concern about disability fraud. If it's coming out of taxes, either state or federal, taxpayers are paying for it or going into debt for it or seeing the currency inflated for it.
They complain that black people are admitted to fancy colleges despite being "less qualified", but reality is the DEI/affirmative action really protected white kids place in the social order, especially the legacy people whose qualifications are marginal. Those kids are being squeezed out by smarter, harder working Asian kids, so those white kids are going to southern prestige schools.
People laugh at culture warriors because of stuff that is targeted at stupid people like the Starbucks "war on Christmas". But there's a more serious/malignant aspect to it as well.
"Better qualified" generally boils down to "people who look like me".
"Fraud" is usually code for "make someone else pay". People make all sorts of passionate appeals to all sorts of moral culpability and scamming, which are mostly bullshit. Specific to the Southeast US, basically that translates to we don't want to have people on social services rolls that the state/locality has a cost share, but we're happy to lobby to make the rules such that the "hand up" is a transition to Social Security Disability, which is funded by not them.