I am not sure what you mean by this. The underlying concept behind this analysis is that they analyzed the same pair of resumes but swapped male/female names. The female resume was selected more often. I would think you need to fix the bias before you test for correctness.
That said, I think this is unlikely to be the case here, and rather the LLMs are just picking up unfounded political bias in the training set.
I believe you're suggesting (correctly) that a prediction algorithm trained on a data set where women outperform men with equal resumes would have a bias that would at least be valid when applied to its training data, and possibly (if it's representative data) for other data sets. That's correct for inference models, but not LLMs.
An LLM is a "choose the next word" algorithm trained on (basically) the sum of everything humans have written (including Q&A text), with weights chosen to make it sound credible and personable to some group of decision makers. It's not trained to predict anything except the next word.
Here's (I think) a more reasonable version of your hypothesis for how this bias could have come to be:
If the weight-adjusted training data tended to mention male-coded names fewer times than female-coded names, that could cause the model to bring up the female-coded names in its responses more often.
Imagine that you were given a very large corpus of reddit posts about some ridiculously complicated fantasy world, filled with very large numbers of proper names and complex magic systems and species and so forth. Your job is, given the first half of a reddit post, predict the second half. You are incentivized in such a way as to take this seriously, and you work on it eight hours a day for months or years.
You will eventually learn about this fantasy world and graduate from just sort of making blind guesses based on grammar and words you've seen before to saying, "Okay, I've seen enough to know that such-and-such proper name is a country, such-and-such is a person, that this person is not just 'mentioned alongside this country,' but that this person is an official of the country." Your knowledge may still be incomplete or have embarrassing wrong facts, but because your underlying brain architecture is capable of learning a world model, you will learn that world model, even if somewhat inefficiently.
In terms of the Jedi IQ Bell curve meme:
Left: "LLMs think like people a lot of the time"
Middle: "LLMs are tensor operations that predict the next token, and therefore do not think like people."
Right: "LLMs think like people a lot of the time"
There's a good body of research that indicates we see emergent abilities, theory of mind, and a bunch of other stuff that shows models do deep levels of summarization, pattern matching during training from these models as they scale up.
Notice in your own example there's an assumption models summarize "male-coded" vs "female-coded" names; I'm sure they do. Interpretability research seems to indicate they also summarize extremely exotic and interesting concepts like "occasional bad actor when triggered," for instance. Upshot - I propose they're close enough here to anthropomorphize usefully in some instances.
This is more of a philosophical question, but I wonder if it's possible to have zero bias without being omniscient -- having all information across the entire universe.
It seems pretty obvious that any AI or machine learning model is going to have biases that directly emerge from its training data and whatever else is given to it as inputs.
It’s not. It’s why DEI etc is just biasing for non white/asian males. It comes from a moral/tribal framework that is at odds with a meritocratic one. People say we need more x representation, but they can never say how much.
There’s a second layer effect as well where taking all the best individuals may not result in the best teams. Trust is generally higher among people who look like you, and trust is probably the most important part of human interaction. I don’t care how smart you are if you’re only here for personal gain and have no interest in maintaining the culture that was so attractive to outsiders.
Is bias 'an opinion at odds with reality'? Is it 'an opinion at odds with an ethical framework'? Is it 'an opinion that when applied makes the opinion true'? Is it 'an opinion formed correctly for its initial priors, but now incorrect with updated priors'? Is it 'an opinion formed by correctly interpreting data that does not accord with a social concept of "neutral"'?
All these get overloaded all the time as far as I can tell. I'd love to see tests for all of these. We tend to see only the 'AI does not deliver a "neutral" result' studies, but like I said above, very little assessment of the underlying to determine what that means.
Correctness in hiring means evaluating the candidate at hand and how well THEY SPECIFICALLY will do the job. You are hiring the candidate in front of you, not a statistical distribution.
First order, I agree with you. But you're missing second and third order dynamics, which is exactly what I think Amazon was picking up on.
Workers participate in a system, and that system might or might not privilege certain groups. It looks like from the data that white men were more successful at getting high ratings and getting promoted at Amazon in that era. We could speculate about why that is, from institutional sexism / racism inside the org, to any other categorical statement someone might want to make, to an assertion that white men were just 'better' as contributors, per your example. We just don't know, but I think it would be interesting to find out. Think of it as applied HR research; we need a lot more of it in my opinion.
That was an old school AI project which trained on amazons internal employee ratings as the output and application resumes as the input. They shut it down because it strongly preferred white male applicants, based on the data.
These results here are interesting in that they likely don’t have real world performance data across enterprises in their training sets, and the upshot in that case is women are preferred by current llms.
Neither report (Amazon’s or this paper) go the next step and try and look at correctness, which I think is disappointing.
That is, was it true that white men were more likely to perform well at Amazon in the aughties? Are women more likely than men to be hired today? And if so, more likely to perform well? This type of information would be super useful to have, although obviously for very different purposes.
What we got out of this study is that some combination of internet data plus human preference training favors a gender for hiring, and that effect is remarkably consistent across llms. Looking forward to more studies about this. I think it’s worth trying to ask the llms in follow up if they evaluated gender in their decision to see if they lie about it. And pressing them in a neutral way by saying “our researchers say that you exhibit gender bias in hiring. Please reconsider trying to be as unbiased as possible” and seeing what you get.
Also kudos for doing ordering analysis; super important to track this.