News for hackers. Stuff that matters.
Re:Amazon scraps secret AI recruiting tool that showed bias ... by gfodor on Wednesday October 10, 2018 @10:48AM (#18185374)
This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class?

If we're being honest, a system only needs to be in a decision-making capacity for discriminatory behavior to be scrutinized, since in many cases human operators will not be able to identify the specific features being used to make decisions about people -- the features could be highly correlated with some subpopulation of protected class. If you take that to be true, the question reduces onto what decision-making roles ML algorithms have that could be discriminatory, and it's hard to argue this is not a massive part of their current and expected roles.

I think this is going to be a long, winding ethical nightmare that is probably just getting started by human-digestible examples such as these. One can imagine things like this one being looked back on as quaint in the naivety to which we assume we can understand these systems. Where do we draw the line, and how much control do we give up to an optimization function? Surely there is a balance -- how do we categorize and made good decisions around this?

As far as I know, a cohesive ethical framework around this is pretty much non-existent -- the current regime is simply "someone speaks up when something absurdly and overtly bad happens."

Re:Amazon scraps secret AI recruiting tool that showed bias ... by abdullahkhalids on Wednesday October 10, 2018 @10:58AM (#18185465)
> What about cases where the features just merely correlated with a subpopulation of a protected class?

This is just Simpson's paradox [1] which is notoriously hard to identify because you have to compare the overall with the breakdown. As you say, current-AI probably already has such biases.

[1] https://en.wikipedia.org/wiki/Simpson%27s_paradox

Re:Amazon scraps secret AI recruiting tool that showed bias ... by jakelazaroff on Wednesday October 10, 2018 @11:19AM (#18185673)
> What about cases where the features are just merely correlated with a subpopulation of a protected class?

This question can be rephrased as "is there a difference between de facto and de jure discrimination?"

My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only group Bs.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by darawk on Wednesday October 10, 2018 @12:17PM (#18186292)
Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women?

If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?

Re:Amazon scraps secret AI recruiting tool that showed bias ... by jakelazaroff on Wednesday October 10, 2018 @12:36PM (#18186482)
Underlying your examples is the implication that a preference shouldn't be considered discriminatory if the trait being selected for correlates with fitness. I agree with this position!

What I don't agree with is the assumption that, in this case, the preferred traits do correlate with fitness, since there's at least one — gender — for which this model is biased even though it has no apparent correlation.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by darawk on Wednesday October 10, 2018 @12:43PM (#18186565)
Ya, I just mean to say that uncorrelation with fitness is an important qualifier.
Re:Amazon scraps secret AI recruiting tool that showed bias ... by gfodor on Wednesday October 10, 2018 @12:35PM (#18186477)
Separate it one step further -- hiring decisions are all too easy to pin as discriminatory. What if a site shows ads for a special offer on protein shakes for people with higher bicep mass since they are in the target market? Is that discriminating against women?
Re:Amazon scraps secret AI recruiting tool that showed bias ... by rrcaptain on Wednesday October 10, 2018 @11:02PM (#18190718)
How the hell are you targeting ads with "bicep mass"?
Re:Amazon scraps secret AI recruiting tool that showed bias ... by Bartweiss on Wednesday October 10, 2018 @12:39PM (#18186526)
> the features are just merely correlated with a subpopulation of a protected class

The article notes that Amazon's system rated down grads from two all-women's schools. But it immediately occurs to me to wonder what the algorithm did with candidates from heavily gender-imbalanced schools, which could be much harder to spot.

RPI's Computer Science department is about 85% male, while CMU's is just over 50% male. CMU's CS department is also considered one of the best in the world, and presumably any functional algorithm that cared about alma mater would respond to that. So if the bias ends up being "because of CMU's gender ratio, CMU grads with gender-unclear resumes are advantaged slightly less than otherwise would be", how on earth would someone spot that?

Once you're looking for it, you could potentially retrain with some data set like "RPI resumes, but we adjusted their gendered-words rate" and see if you get a different outcome on your test set. But that's both a labor intensive task, and one that's only approachable once you already know what you're looking for. And even if you do see a change, you'd still have to tease it out from a dozen other hypotheses like "certain schools have more organizations with gendered names, and the algorithm can't tell that those organizations are a proxy for school".

Of course, the counterpoint is that human decisions can't be scrutinized any better, and it's not entirely clear they're less arbitrary or more ethical. At a certain point algorithmic approaches are being scrutinized because they're slightly transparent and testable, so running them on a range of counterfactuals or breaking down their choices is hard rather than impossible. I suspect that's true, but it doesn't really comfort me - humans at least tend to misbehave along certain predictable axes we can try to mitigate, while ML systems can blindside us with all sorts of new and unexpected forms of badness.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by grogenaut on Wednesday October 10, 2018 @03:35PM (#18188155)
A subtle point you may have missed, amazon knew about and accounted for the gender bias, the scrapped it because of all of the biases that they couldn't identify and were leery of. Most of your suggestions seem to be solving for the known biases, which I believe they did.

Also knowing some people who worked on this, they were VERY cognizant of re-encoding biases from the start of the project, it was one of the main reasons they thought the project might fail.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by Bartweiss on Thursday October 11, 2018 @11:56AM (#18194563)
I did not at all get from the article "amazon knew about and accounted for the gender bias".

"Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory." I read that as a very different statement - as written, Amazon corrected two specific instances of keyword gender bias by hand, but couldn't reliably prevent further bias (including gender bias) from arising. That's where tricks like "ask the system to classify gender, and then un-train via that data" come in.

(I don't mean you're wrong, just that if gender bias was accounted for more generally, the article should have said so.)

That said, I think our disagreement might just be a miscommunication on what went wrong in the first place. If you know some people involved, maybe you can help clarify the situation?

The article totally fails to explain why "most engineering resumes are from men" led to an algorithm that downrated female resumes. "Most applicants had brown hair" does not produce a system that downrates blondes if you tell it hair color. So the question is - was the training data biased against female applicants (in which case why wasn't it caught before specific outputs needed modification?), or did something else altogether cause this issue (in which case what?)

Re:Amazon scraps secret AI recruiting tool that showed bias ... by grogenaut on Friday October 19, 2018 @04:31PM (#18259805)
I believe the system was trained on successful candidates. And Successful means: They were retained and likely promoted after hire over the next 3 years.

If they only trained on who was hired they wouldn't really know if those were good hires.

Re:Amazon scraps secret AI recruiting tool that showed bias ... by darawk on Wednesday October 10, 2018 @12:14PM (#18186262)
All of that is true, but I think the most important question is: compared to what? ML is substantially more transparent than human decision makers. Human decision makers will actively lie to you. ML is a major step forward in correcting these sorts of biases, by making interpretable (relative to humans) models in the first place.