> it’s being used to job screen applicants even though we have proven that AI models still suffer from thing like racial bias

I bet they also suffer from other biases that are harder to detect and maybe some biases we can't even imagine and thus control for.

Biases like statistically significant correlated features.

Kind of like humans in that regard

Except being able to process 1e9 more applications, thus massively more biased and discriminatory.

So.....,just like humans?

That's the point though. It's illegal for humans to do things during hiring that are not illegal for AI to do as it stands now (all sorts of discrimination). We want to at least level the playing field.

There’s no exemption from prosecution for racial bias because it was done by an algorithm. If a company uses such an algorithm, AI or not, they will be open to prosecution for it.

The differential nugget is that LLMs and similar tech are auditable. You can audit the models, test for bias etc in a way that scales.

how do you audit the loaded die at the end of all llms?

Are there agreed upon neutral auditors or common processes for auditing models? If not, maybe what we need is (gasp!) some governmental oversight.

Humans can be asked to explain their decissions and personally take accountability for them. Even if AI processes were less biased, using them introduces the opacity of relying on models that simply can't be fully intuited or explained. For life-altering decisions, merely performing better than a human is not the benchmark.

there maybe better examples of why AI biases provide deep systemic problems but: CV screening? are contemporary LLM really worse that previous screening tech and processes?

You can combat bias in a team of people, but in LLM you may not even know it exists.

How can you combat bias in a team of humans if humans are susceptible to bias? How do you even know what neutral is if it's not a physically measurable quantity?

This sounds like an argument for never educating people and doing nothing about the problem.

Just because you graduated from a CS program doesn't mean you will always write bug free code using the ideal algorithm and design pattern.

But you know what you should be striving for, can more readily identify issues, and maybe sometimes you actually are perfect.

> This sounds like an argument for never educating people and doing nothing about the problem.

I feel like it's really the opposite.

There is a major problem with using AI to screen CVs, which is that AI is bad at screening CVs. It excludes good candidates and offers mediocre ones, and opens up a bunch of hacks the equivalent of black hat SEO, which gives an undesired advantage to people inclined to dirty tricks.

But "it's racist" is a political hot button, so before people can start to point that out, someone scrambles to push the hot button and suck all the oxygen out of the room. Then the purveyors of snake oil can munge the algorithm until it passes the naive oversimplified racial bias test someone made up (probably making it even worse at its intended purpose), and then claim "the problem" is solved, as if that was the main and only problem.

They send resumes with the same accomplishments and names that indicate race and observe the relative degrees of acceptance.

Everytime they do this test, considerable bias is found to still exist.

remove names then. problem solved.

There are still plenty of proxies that can amount to the same thing. Not a lot of white graduates of HBCUs, for instance. Are we removing the names of all schools, too?

if you dont want to favor ivy leagues that would be the next thing to remove yes

Still using resumes, have an ai interview for knowledge and communication skill.

Like many others, we've built a product to do this.

I think it gives a much higher signal on candidates than a resume, and is more "blind" in terms of bias than a human.

It's based on voice messages. Open to all here if anyone wants to take it for a spin. https://candit.net

How do you get an unbiased evaluation of someone with an accent that differs from your training set? It took about a decade for my friends with Indian English accents to be able to use voice controls in luxury cars. And honestly young people from India have a much more American accent than my generation.

It‘s 2024. There are decades or research with a lot of data, there exist best practices and products that help combating the bias. The toolkit available to HRs is rich and there’s plenty of things that work. E.g. I have absolutely no problem hiring women in tech roles, because I simply adjusted my process. I heard complaints from other engineering leaders that they want to increase diversity but struggle to find candidates: they do not realize that they have bias in job descriptions, they apply same screening criteria and do the post-interview evaluation the same way regardless of candidate gender or background. Many people can do the job, but they communicate it differently and it is important to account for that when you see the CV or talk to them.

Isn't not knowing that bias exists just the same problem that the average HR department has? You solve that by educating people about biases. If anything that should be easier with LLMs because you're not asking people to alter their sense of self ("I wouldn't do that!"), you're just informing them of technical limitations.

I don’t understand your argument. The whole purpose of AI-driven recruitment is to reduce human effort by doing some work for them. If AI due to existing bias reduces diversity in your pool of candidates, what can you do with it? You either do not notice the bias and just think that those people are indeed the best fit, you willingly accept the bias („But it reduces our hiring costs! We deal with diversity later, when this technology matures!“), or you go to see the unfiltered stream and put such an amount of work into it that makes AI useless. HR departments are not the people who will build their own solution, they will buy it from a third party and they won’t have control over or the budget for the fine-tuning.

of course you can know it exists. you just need to observe inputs and outputs and compare with precious non LLM ways.

You do not understand how HR works then. They are not LLM researchers and just apply the available toolkit, e.g. organizing regular training for hiring managers. When they buy AI-driven solution from a third party, they have no tools neither to detect if it’s just a statistical fluctuation or there’s a bias they need to find a workaround for, nor to act on this information in a cost-efficient way. Most likely they won‘t even have enough data to reach any statistical significance. The vendor in theory could do it, but most of them are startups now and they do not have diversity in their OKRs, at least not now. It’s not the UVP of their product.

Of course they could be. A company that doesn't want a racial bias won't intentionally filter on names, but might accidentally deploy an LLM that can discern race from name.

The humans will do it unconsciously if they can see the names.

It is like orchestras used to use auditions where the judges could see the people; when they went to auditions where they couldn't see the people, the number of women being hired went right up.

See, this is the problem with just deciding to invent your own facts.

There is a famous study on this topic, usually presented in much the way you describe. But usually people are more careful not to lie about the results. The rate of women being hired went down, not up. The "success" for women was contained to advancement through particular audition rounds.

https://archive.is/xmvp2

The original paper is not exactly a model of investigative integrity:

>> Women are about 5 percentage points more likely to be hired than are men in a completely blind audition, although the effect is not statistically significant. The effect is nil, however, when there is a semifinal round, perhaps as a result of the unusual effects of the semifinal round.

Are you citing a weird racist blogger on medium that disagrees with the stated conclusion of the paper? Is that how science works now?

Abstract from the original paper:

> A change in the audition procedures of symphony orchestras--adoption of "blind" auditions with a "screen" to conceal the candidate's identity from the jury--provides a test for sex-biased hiring. Using data from actual auditions, in an individual fixed-effects framework, we find that the screen increases the probability a woman will be advanced and hired. Although some of our estimates have large standard errors and there is one persistent effect in the opposite direction, the weight of the evidence suggests that the blind audition procedure fostered impartiality in hiring and increased the proportion women in symphony orchestras.

https://www.aeaweb.org/articles?id=10.1257/aer.90.4.715

You'll get more accurate results if you look at the paper's conclusion, not the abstract.

The whole point is that the paper doesn't actually support any of the claims, which it's fairly open about -- that's those "large standard errors".

They did not in fact find that the screen increases the probability of a woman being hired. They found that it increased the probability of a woman passing the final round of auditions. There's more than one round.

It's up to you, I guess, whether you want to follow the original authors to their conclusion that the semifinal round is fundamentally unlike the final and quarterfinal audition rounds. I wouldn't.

The paper is meritless, but even it's actual worthless findings aren't in the direction that people like to claim. Here's Andrew Gelman: https://statmodeling.stat.columbia.edu/2019/05/11/did-blind-...

> First, some equivocal results:

> This is not very impressive at all. Some fine words but the punchline seems to be that the data are too noisy to form any strong conclusions.

> Huh? Nothing’s statistically significant but the estimates “show that the existence of any blind round makes a difference”? I might well be missing something here. In any case, you shouldn’t be running around making a big deal about point estimates when the standard errors are so large. I don’t hold it against the authors—this was 2000, after all, the stone age in our understanding of statistical errors. But from a modern perspective we can see the problem.

> Anyway, where’s the damn “50 percent” and the “increases by severalfold”? I can’t find it. It’s gotta be somewhere in that paper, I just can’t figure out where.

> Pallesen’s objections are strongly stated but they’re not new. Indeed, the authors of the original paper were pretty clear about its limitations. The evidence was all in plain sight.

> For example, here’s a careful take posted by BS King in 2017:

>> Okay, so first up, the most often reported findings: blind auditions appear to account for about 25% of the increase in women in major orchestras. . . . [But] One of the more interesting findings of the study that I have not often seen reported: overall, women did worse in the blinded auditions. . . .

If you read the paper, you'd notice the problems.

But you might not want to run the risk of, um, "racism" that seems to be inherent in reading a paper on the impact of blind auditions on women.

The racism was more the guy having very strong opinions about black people's IQs and immigrants destroying western civilization on twitter (plus being against COVID vaccines and human rights and other stuff that suggest you are not a level-headed truth seeker).

Quote from your second, more respectable, link:

> Pallesen’s objections are strongly stated but they’re not new. Indeed, the authors of the original paper were pretty clear about its limitations.

When I read the original papers, like your second link, I find a level headed paper working with limited data they had available.

I would suggest that the annoyance is with pop science retelling at best, or just a quack selling anger online to fuel his weird obsessions at worst.

A human might, but a dumb filter wouldn't.

You realize this is a huge problem right? Even if that was my only complaint (it’s not!) that is 100% not acceptable. If it’s doing it with CVs it’s doing it with other things in other scenarios.

Another is that a health insurance company was caught using AI to determine if a claim should be denied or not which lead to a scandal as a whistleblower leaked the practice, as it was wrought with errors and ethical concerns

It's almost guaranteed there are people trying to sell an AI version of Robodebt right now to the Australian government, even though the last (non-AI) version of it was an absolute cluster fuck:

https://en.wikipedia.org/wiki/Robodebt_scheme

Other governments around the world have done similar (non-AI) things in the past with similar terrible results. They'll likely try an AI version of things too in the near future, just because "AI" apparently solves all the problems. Ugh.

I think a worse feature for them to have is consistency. Imagine being someone that has fallen through the cracks of software a significant number of employers are using... Basically soft-locked out of employment with no recourse.

I'd argue consistency in discrimination is easier because it makes it easier to detect and account for.

A range of different people interviewing are going to be alot harder to pin down.

Discrimination in this sense is a statistical property. If Jimbob gets rejected everywhere as an individual it isn't really discrimination but that doesn't mean it isn't an issue.

I would argue that we already have experienced enough of the downsides of "AI" that there is reasonable cause for concern.

The implications of deepfakes and similar frauds alone are potentially devastating to informed political debate in democracies, safe and effective dissemination of public health information in emergencies, and plenty of other realistic and important trust scenarios.

The implications of LLMs are potentially wonderful in terms of providing better access to information for everyone but we already know that they are also capable of making serious mistakes or even generating complete nonsense that a non-expert user might not recognise as such. Again it is not hard to imagine a near future where chat-based systems have essentially displaced search engines and social media as the default ways to find information online but then provide bad advice on legal, financial, or health matters.

There is a second serious concern with LLMs and related technologies, which is that they could very rapidly shift the balance from compensating those who produce useful creative content to compensating those who run the summary service. It's never healthy when your economics don't line up with rewarding the people doing the real work and we've already seen plenty of relevant stories about the AI training data gold rush.

Next we get to computer vision and its applications in fields like self-driving vehicles. Again we've already seen plenty of examples where cars have been tricked into stopping suddenly or otherwise misbehaving when for example someone projected a fake road sign onto the road in front of them.

Again there is a second serious concern with systems like computer vision, audio classification, and natural language processing and that is privacy. It's bad enough that we all carry devices with cameras and microphones around with us almost 24/7 these days and the people whose software runs on those devices seem quite willing to spy on us and upload data to the mothership with little or any warning. That alone has unprecedented implications for privacy and associated risks. With the increased ability to automatically interpret raw video and audio footage - with varying degrees of accuracy and bias of course - that amplifies the potential dangers of these systems greatly.

There is enormous potential in modern AI/ML techniques for everything from helping everyday personal research to saving lives through commoditising sophisticated analysis of medical scans. But that doesn't mean there aren't also risks we already know about at the same kind of scale - even without all the doomsday hypotheticals where suddenly a malicious AGI emerges that takes over the universe.

The issue is, the regulations are tailored to address any of those concerns, some of which may not even be solvable through regulation at all:

> The implications of deepfakes and similar frauds alone are potentially devastating to informed political debate in democracies, safe and effective dissemination of public health information in emergencies, and plenty of other realistic and important trust scenarios.

The horse is out of the barn on this one. You can't stop this by regulating anything because the models necessary to do it have already been released, would continue to be released from other countries, and one of the primary purveyors of this sort of thing will be adversarial nation states, who obviously aren't going to comply with any laws you pass.

> The implications of LLMs are potentially wonderful in terms of providing better access to information for everyone but we already know that they are also capable of making serious mistakes or even generating complete nonsense that a non-expert user might not recognise as such.

Which is why AI summaries are largely a gimmick and people are figuring that out.

> they could very rapidly shift the balance from compensating those who produce useful creative content to compensating those who run the summary service.

This already happened quite some time ago with search engines. People want the answer, not a paywall, so the search engine gives them an unpaywalled site with the answer (and gets an ad impression from it) and the paywalled sites lose to the ad-supported ones. But then the operations that can't survive on ad impressions lose out, and even the ad-supported ones doing original research lose out because you can't copyright facts so anyone paying to do original reporting will see their stories covered by every other outlet that doesn't. Then the most popular news sites become scummy lowest-common-denominator partisan hacks beholden to advertisers with spam-laden websites to match.

Fixing this would require something along the lines of the old model NPR used to use, i.e. "free" yet listener-supported reporting, but they stopped doing that and became a partisan outlet supported by advertising. The closest contemporary thing seems to be the Substacks where most of the stories are free to read but you're encouraged to subscribe and the subscriptions are enough to sustain the content creation.

The AI thing doesn't change this much if at all. A cheap AI summary isn't going to displace original content any more than a cheap rephrasing by a competing outlet does already.

> Next we get to computer vision and its applications in fields like self-driving vehicles. Again we've already seen plenty of examples where cars have been tricked into stopping suddenly or otherwise misbehaving when for example someone projected a fake road sign onto the road in front of them.

But where does the regulation come in here? When it does that it's obviously a bug and the manufacturers already have the incentive to want to fix it because their customers won't like it. And there are already laws specifying what happens when a carmaker sells a car that doesn't behave right.

> Again there is a second serious concern with systems like computer vision, audio classification, and natural language processing and that is privacy.

Which is really almost nothing to do with AI and the main solutions to it are giving people alternatives to the existing systems that invade their privacy. Indeed, the hard problem there is replacing existing "free" systems with something that doesn't put more costs on people, when the existing systems are "free" specifically because of that privacy invasion.

If a government wants to do something about this, fund the development of real free software that replaces the proprietary services hoovering up everyone's data.

Let’s stipulate that all you said was true. How is EU regulation suppose to prevent that? Are they going to stop open source models from being used in Europe? Are they going to stop foreign adversaries from using deep fakes?

It’s just like trying to restrict DVD encryption keys from being published or 128 bit encryption from being “exported” in browsers back in the car.

> The implications of LLMs are potentially wonderful in terms of providing better access to information for everyone but we already know that they are also capable of making serious mistakes or even generating complete nonsense that a non-expert user might not recognise as such. Again it is not hard to imagine a near future where chat-based systems have essentially displaced search engines and social media as the default ways to find information online but then provide bad advice on legal, financial, or health matters.

I think a bigger concern is LLMs providing deliberately biased results and stating them as fact.

Problem is most innovation is in using existing models in new ways. You can't expect these most people to train their own models.

Regulating it the way you say just means "zero innovation!".

> For example, it’s being used to job screen applicants even though we have proven that AI models still suffer from thing like racial bias.

Can't we just say racism is illegal, and if a company uses an AI to be racist, they get fined the same way they would if they were racist the old fashioned way?

Look up the Dutch tax return scandal where the Dutch tax arm of the government (‘IRS’) used machine learning to identify fraud but it turned out to be very racially biased and it uprooted thousands of families with years of financial struggles and legal battles.

See https://en.m.wikipedia.org/wiki/Dutch_childcare_benefits_sca...

or the royal mail horizon scandal. although the software not called an "ai" afair

"Just", no.

Fence at the top of a cliff (make sure the AI is unbiased and can be fixed when it turns out it is) vs. Ambulance at the bottom (letting people sue if they think the machine is wrong).

When you make specific methods illegal, it tends to lead to loopholes. If you just make the result illegal then nobody can try and get around it by using a slightly different system design.

Also true.

But which way around does that apply? Racism is a concrete example of what AI may incorrectly automate, it's not the only bias that AI can have — any correlation in the training data will be learned, but not necessarily causation — and the goal of these laws is to require the machines to be accurate, unbiased, up to date, respectful of privacy etc., with "not racist" as merely one example of why that matters.

(Also the existing laws on racial equality were not removed by GDPR; to the previous metaphor, the fence being better doesn't mean you can fire the ambulance service).

> it’s being used to job screen applicants

Any idea what software is being used ?

Just an example:

https://hirebee.ai/

How do you know the race?

I've seen some places collect that information which is wild. But you can decline.