News for hackers. Stuff that matters.
Re:Amazon scraps secret AI recruiting tool that showed bias ... by ncallaway on Wednesday October 10, 2018 @10:39AM (#18185261)
I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got).

Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

Re:Amazon scraps secret AI recruiting tool that showed bias ... by michaelt on Wednesday October 10, 2018 @11:00AM (#18185504)

  Could this be a direct indicator of a powerful
  subconscious bias in Amazon's existing hiring
  process?
Maybe - but maybe not.

Imagine a company with 2 men in HR, 2 women in HR, 40 men in engineering, and 10 women in engineering. That's with gender-blind hiring, reflecting only the 4:1 ratio of male to female CS graduates.

If you picked a random male hire, there's a 40/42=95% chance they're an engineer whereas if you picked a random female hire, there's a 10/12=83% chance they're an engineer.

Thus if you look over all hires' CVs, due to Bayes’ law the dataset says being male increases the conditional probability you meet engineering hiring requirements - and the ML system picks up on that.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by skohan on Wednesday October 10, 2018 @11:24AM (#18185728)
So it depends how you set up the experiment right? I would assume the question you are posing to AI is not how much the resume in question resembles the set of resumes of hired engineers, but rather: given a resume, what is the probability that candidate will eventually be hired?

So the classification function should take into account the resumes of rejected engineers, rather than the pool of resumes of hired employees at Amazon. If someone is seeking a position as an engineer, it is not relevant how much their resume resembles that of HR people, but it is very relevant how much it resembles that of rejected engineering candidates.

If that's the case, then something like having the phrase "women's chess club" in one's resume should not be a meaningful factor for the classifier unless it disproportionately leads to rejection in the current process.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by sanxiyn on Wednesday October 10, 2018 @11:12AM (#18185616)
Why would one train on "all hires CVs"? It'd be "engineering CVs", moreover it'd be "engineering applicants CVs", not "engineering hires CVs".
Re:Amazon scraps secret AI recruiting tool that showed bias ... by Bartweiss on Wednesday October 10, 2018 @12:55PM (#18186690)
In the article I noticed: "Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs"

That language is a bit ambiguous, it could just mean that the algorithm failed on a wide variety of jobs beyond engineering. But another reading suggests that the algorithm was not asked "is this person a good fit for this role" but instead "what, if anything, is this person qualified for?"

If that's the case, then the problem starts to make more sense: the algorithm learned a correlation between male-sounding resumes and being hired for engineering roles. That could produce a biased approach even if the decisions in the training data were gender neutral but position-specific. Of course, it would also mean that an Amazon ML team trained an algorithm with inputs that didn't match to its eventual task, and makes me wonder what they used as a test set...

(Anecdotally, Amazon spent quite a while recruiting me for SysEng work I'm wildly unqualified for and uninterested in, even suggesting a switch to applying for that team when I was already in the funnel for something I'm more qualified at. When my resume eventually made it to a syseng engineer, they were rightly baffled that I had landed on in their stack, giving me the sense that something was screwy with how Amazon decides who heads towards which role.)

Re:Amazon scraps secret AI recruiting tool that showed bias ... by michaelt on Wednesday October 10, 2018 @11:22AM (#18185707)
Because you were trying to build a system that would perform CV filtering for your entire company, and you figured Deep Learning would just kinda take care of everything.

You're right you'd want to look at applicants' CVs - I skipped over that to make the numbers readily comprehensible.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by megy on Wednesday October 10, 2018 @10:10PM (#18190564)
Because we are guessing at the problem, we don't have complete data.
Re:Amazon scraps secret AI recruiting tool that showed bias ... by nkassis on Wednesday October 10, 2018 @01:46PM (#18187243)
If they naively just feed the entire dataset without some inspection maybe. But I doubt the people working on this model stopped at the most basic level. What you describe is a common class imbalance issue. I would expect that they have accounted and addressed (at a minimum oversampling the less represented class for example) for this issue while working on the model.

So I doubt it's enough to explain their issue here. I agree that we can't really take any conclusion of their broader hiring patterns from this experiment.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by throwawaymath on Wednesday October 10, 2018 @10:44AM (#18185330)
> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon.

Without significant oversight and manual tuning, any training dataset based wholly or in part on current employees is going to demonstrate a bias against women, because there are strictly fewer women. Moreover it's likely that (for a variety of reasons, both intrinsic and extrinsic) fewer women succeed in the interview process as a ratio compared to the number who actually apply.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by Spooky23 on Wednesday October 10, 2018 @11:13AM (#18185617)
I spent a number of years working in tech organizations where the male:female ratio was 60:40. Now I work in a similar org with the same rules where it's more like 85:15.

The difference is that old place transitioned administrative staff to IT roles in the 90s/early 2000s when more things were computerized. Those admins, financial analysts, program analysts were more likely to be female and had degrees in liberal arts, accounting, business/finance, etc.

In the newer place, they filtered based on computer-related degree upon hiring. That automatically excludes many women. Once hired, female candidates advance as well or better.

Anecdotally, I've hired interns in recent years with no tech-specific qualifications as an experiment. If you select for "smart and gets things done" I don't see much of a disadvantage for many roles. You get some duds too, but it wasn't as dramatic a difference as I expected.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by hellogoodbyeeee on Wednesday October 10, 2018 @11:46AM (#18185986)
Do FAANG companies train software developers? I think they are big enough and opinionated enough organizations that they might be better at training a "FAANG quality developer" in a year or so than current University system can in four years.

I wish companies were more willing to train willing applicants instead of trying to interview for capabilities.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by mcguire on Wednesday October 10, 2018 @11:14AM (#18185630)
"...any training dataset based wholly or in part on current employees is going to demonstrate a bias against women,..."

"Demonstrate" may be the wrong word. How about "encode". And in the future, "enforce".

[Edit] On second thought, it would seem like there would be a way to filter out a "raw proportions bias" (like 80% of the resumes in the data set are male) before training.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by KevinEldon on Wednesday October 10, 2018 @10:47AM (#18185359)
Maybe, but it seems more likely that the model just didn't work.

From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"

Re:Amazon scraps secret AI recruiting tool that showed bias ... by WalterSear on Wednesday October 10, 2018 @10:51AM (#18185398)
Sounds like they've successfully emulated their real world interview process.
Re:Amazon scraps secret AI recruiting tool that showed bias ... by sanxiyn on Wednesday October 10, 2018 @11:08AM (#18185573)
Indeed.

I am quite curious about details of the model. For example, the single largest contribution to real world interview process variability is interviewer (for resume screening, who screened that resume, etc.). Wouldn't it be possible to code interviewer as categorical variable and separate resume-intrinsic? effect and interviewer effect? They must have tried this, haven't they?

Re:Amazon scraps secret AI recruiting tool that showed bias ... by throwawaymath on Wednesday October 10, 2018 @10:54AM (#18185428)
Those aren't mutually exclusive. Technically speaking, the model can return results "almost at random" and still demonstrate a bias against any particular attribute if that bias is evident in the underlying training dataset.

If there are strictly fewer women in the underlying training set, the model can still return something resembling a uniform distribution of candidates while exacerbating the diminished representation of women.

To give a concrete example: you have a bag of blue dice and red dice. There is a supermajority of blue dice in the bag. Your algorithm selects a single die out of the bag on every iteration. The output sequence of dice numbers appears uniform, but there are more blue dice than red dice in the output sequence.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by TangoTrotFox on Wednesday October 10, 2018 @11:26AM (#18185748)
Step into your engineering or computer science department and walk into any upper division class and count the females and count the males. That some companies have upwards of 20% females is more likely indicative of extreme bias in hiring as you're not going to find even remotely close to 20% females there. Enter in most measures of competence and you'll find the division is no different. E.g. - if females were being disproportionately hired because of disproportionately positive performance then this might not be an issue, but there seems to be no evidence for that whatsoever.

This view that the only thing holding people back is some sort of social or systemic bias seems to be based on nothing except ideology. Incidentally, it's an ideology I also used to hold. Like a good egalitarian I pushed my wife away from sociology and into majoring in CS. She did perfectly well, as did I. More than a decade later she works with people and I work with code. I've no regrets there, but it's not so clear that my persuasion was really the best idea.

Norway is another interesting example here. It is considered by many to be the most gender equal location in the world. Yet you'll still find that nurses are primarily female, doctors are primarily male, and all other 'stereotypical' divisions present in most all developed nations. They tried to change these divisions and with extensive effort were able to effect a roughly constant change in some fields. But again, once that push was relinquished things went just about identically to as they were in very short order. Ultimately we're flexible enough that you can manage to fit a square peg into a round hole at times, but once you stop squeezing that peg goes back to what it wants to be.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by mcguire on Wednesday October 10, 2018 @12:45PM (#18186588)
"I've no regrets there, but it's not so clear that my persuasion was really the best idea."

Does she make more money than a similar person with a degree in sociology?

Re:Amazon scraps secret AI recruiting tool that showed bias ... by hellogoodbyeeee on Wednesday October 10, 2018 @11:42AM (#18185927)
Right, so the "AI" is not finding the best candidates, but instead finding candidates that succeed in the old hiring process. What I'd like to see (though I'm sure this would get me laughed out of a manager's office) would be a certain fraction of random hires. Then we could train the model on success statistics based on 60,90,120 day performance (or longer!).

At the least, they should interview a certain subset of randomly chosen applicants or else the feedback loop from the interviewing process and the AI is going to grow tighter and tighter.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by elfakyn on Wednesday October 10, 2018 @10:45AM (#18185340)
That gives rise to a very interesting concept: ML-based bias assessments. If you take some real-life hiring data (or other applications such as sentencing or generally human behavior data) and train the AI on it, then run it through a bunch of tests to see whether there's bias, that can reveal trends in the underlying training data.

I can't imagine this not already being a thing, but I haven't really heard of people using this method.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by bloak on Wednesday October 10, 2018 @11:31AM (#18185793)
I don't think you can detect "bias" by running "a bunch of tests". "Bias" is a very slippery concept and is probably essentially subjective. When people say an algorithm is "biased" what they seem to mean is that when the judgements of the algorithm are compared with the judgements of a committee of fair-minded and diligent humans then the number of positive outcomes for members of some fashionable minority that we care about is less that what it was with the human judges. It's hard to automate that. And in any case, if you manipulate the algorithm until it "passes" a test like that then you might not really have improved it: when you turn a measure into a target it ceases to be a good measure.