News for hackers. Stuff that matters.
Re:Sent to Prison by a Software Program’s Secret Algorithms by crb002 on Monday May 01, 2017 @05:06PM (#14241593)
You could do automatic differentiation. Take each input, race, sex, age, ... and see how the output changes. If one field has a huge swing it might be a good basis for claiming variable X based discrimination.
Re:Sent to Prison by a Software Program’s Secret Algorithms by BearGoesChirp on Monday May 01, 2017 @05:22PM (#14241774)
Sex would have a major impact. At least in the US our justice system is extremely sexist, far more than it is racist. I once saw sex offender guidelines that automatically put the default sex offender sentence as medium for males and low for females (the default was used before an official evaluation was completed, and I would bet some money that the official evaluations discriminated based on sex).

While I don't disagree our legal system is also racist, the racism pales in comparison to the current level of sexism.

Re:Sent to Prison by a Software Program’s Secret Algorithms by dTal on Monday May 01, 2017 @06:06PM (#14242152)
That's not a bug, it's a feature! User jawns posted some of their marketing copy elsewhere in this thread:

"COMPAS offers separate norms for males, females, community and incarcerated populations"

And they call that "defensible decisions"! In other words, they want to make sure the software doesn't rock the boat. If there's a bias, they want to preserve it so it doesn't look like their algorithms are "weird".

Re:Sent to Prison by a Software Program’s Secret Algorithms by a3n on Monday May 01, 2017 @06:30PM (#14242334)
> In other words, they want to make sure the software doesn't rock the boat. If there's a bias, they want to preserve it so it doesn't look like their algorithms are "weird".

Isn't that what judges do when they follow precedent?

Re:Sent to Prison by a Software Program’s Secret Algorithms by dragonwriter on Monday May 01, 2017 @07:38PM (#14242765)
No, and judges don't always follow precedent (they theoretically have to when it is binding, but no precedent is binding on the highest court, even it's own.)

When they follow precedent, they are not matching outcomes to outcomes to avoid looking wierd, they are applying what was previously established as the formal decision rule.

That's a very different thing that attempting to preserve biases that may not even reflect the formally-annoubced decision rules in the cases that produced them, but may represent silent deviations from the formal rules.

Re:Sent to Prison by a Software Program’s Secret Algorithms by cryptarch on Monday May 01, 2017 @07:19PM (#14242657)
Legal precedents tend not to explicitly single out one gender, race or religion.
Re:Sent to Prison by a Software Program’s Secret Algorithms by a3n on Monday May 01, 2017 @05:19PM (#14241749)
And if there's not a small number of perturbations, then it's the proprietary combination of a large number of small, individual perturbations. But either way, since it's proprietary, it's not practically refutable.

Not to mention you'd never get access to do the differentiation, as it would rely on access to the proprietary process; the company would have to be compelled to cooperate.