News for hackers. Stuff that matters.
Re:Justice.exe: Bias in Algorithmic sentencing by gravypod on Sunday May 07, 2017 @06:04PM (#14287957)
> Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group.

Can you break this down more? It doesn't make sense to me. If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program.

Unless the programmers are changing the datasets then I don't see how this makes sense.

Re:Justice.exe: Bias in Algorithmic sentencing by woodruffw on Sunday May 07, 2017 @06:16PM (#14288020)
In the case you presented, the biases are in the dataset itself.

Arrest statistics in the US are heavily skewed by race. If you were to take a dataset of all US arrests between 1900 and 2000 and ask which populations are most likely to "commit crimes" (i.e., be arrested), you'd get racially biased model without recording why it's biased (discrimination, enduring poverty, minimum sentencing, &c).

Re:Justice.exe: Bias in Algorithmic sentencing by mattnewton on Sunday May 07, 2017 @07:38PM (#14288307)
Dead sibling comment asks a common questions I hear: why can't we just not tell the algorithm race? And the answer is that, with enough data points, if there is a statistical bias in the dataset, the algorithm will likely learn a substitute for race in the other features. For the sake of argument, perhaps an interaction between home address, annual income and model of car driven is predictive of race, and race is predictive of recidivism in the dataset- then the algorithm will learn this cross of features is predictive of recividism, even though we would like all of them to ideally be irrelevant to sentencing.
Re:Justice.exe: Bias in Algorithmic sentencing by gravypod on Sunday May 07, 2017 @07:49PM (#14288337)
> home address, annual income and model of car driven is predictive

Why the hell would those be factors? The factors of the case should be things that actually matter in the case.

A rich person who murders someone should get the same sentence as a poor person who murders someone.

A black person who murders someone should get the same sentence as a white person who murders someone.

Adding those factors would be insane in the first place. If you're adding crazy things like that you might as well add factors like "Can Juggle" and "Can Burp the Alphabet" because things like that should have just as much to do with sentencing as what kind of car you drive or where you live.

Re:Justice.exe: Bias in Algorithmic sentencing by jcranmer on Sunday May 07, 2017 @08:04PM (#14288398)
One example that's going to come up is "Has the person previously been convicted of a crime?" That question is strongly correlated with race, and yet is strongly arguable to be relevant.

Model of car is hard to say that it's useful for predictive value, but home address (=which neighborhood did you grow up in, e.g., are you in Cabrini-Green or are you Gold Coast?) and annual income (poor people are more likely to commit crimes than rich people) definitely are going to be queried inputs. Particularly if the question is things like "what should I set bail at?"

Re:Justice.exe: Bias in Algorithmic sentencing by gravypod on Sunday May 07, 2017 @08:15PM (#14288432)
> "Has the person previously been convicted of a crime?" That question is strongly correlated with race, and yet is strongly arguable to be relevant.

Past behavior is a good predictor of future behavior in my experience. That's more than doubled when it's recurring behavior.

> home address (=which neighborhood did you grow up in, e.g., are you in Cabrini-Green or are you Gold Coast?) and annual income (poor people are more likely to commit crimes than rich people) definitely are going to be queried inputs

Why?

> Particularly if the question is things like "what should I set bail at?"

That's still the job of the judge and there is law at hand for how those values are calculated. The computers won't be involved with that (I'd assume) just sentencing based on previous case decisions.

Re:Justice.exe: Bias in Algorithmic sentencing by pdkl95 on Sunday May 07, 2017 @08:51PM (#14288554)
> Past behavior is a good predictor of future behavior in my experience.

That question includes the past behavior of a lot more people than just the person with a criminal conviction.

> recurring behavior

Institutional racism has recurred a lot in the justice system.

Re:Justice.exe: Bias in Algorithmic sentencing by mattnewton on Sunday May 07, 2017 @08:36PM (#14288505)
I agree they shouldn't be factors, and my example is a bit contrived, but many do include socioeconomic and location based factors, like this reply by the authors of a paper on one recidivism calculator (who argue here why socioeconomic factors are included) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5047350/
Re:Justice.exe: Bias in Algorithmic sentencing by c22 on Monday May 08, 2017 @12:09AM (#14289184)
Of course the factors chosen are just another place to introduce bias.
Re:Justice.exe: Bias in Algorithmic sentencing by dilemma on Sunday May 07, 2017 @08:59PM (#14288576)
>In the case you presented, the biases are in the dataset itself.

The code that is going to handle the data is also biased in some direction. Its actions is value-based, coming from that of the creators of the algorithm. This is why all code must be open and publicly auditable if used in law enforcement, health care and insurance, etc.

Re:Justice.exe: Bias in Algorithmic sentencing by dante821 on Sunday May 07, 2017 @07:21PM (#14288247)
Can't that problem be solved by simply not telling the machine the race of the person who's being judged?
Re:Justice.exe: Bias in Algorithmic sentencing by randomtask on Sunday May 07, 2017 @07:54PM (#14288361)
As with anything, it's hard to generalise without oversimplifying, but here goes. You don't generally just have a data set and a machine learning algorithm that somehow magics outputs from a data set. Usually decisions have to be made by people either in training the model, selecting variables that are included in a model, etc.

Here's a simple example. Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know). We have to tell the algorithm whether a given article in the training set is fake or legitimate, otherwise how would it know? Clearly this will reflect the views of whoever is tagging the articles. When we run the model on a training set we need to score how well it did, again this will reflect the opinion of the person doing the scoring.

For a real example: https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-....

Re:Justice.exe: Bias in Algorithmic sentencing by gravypod on Sunday May 07, 2017 @08:11PM (#14288417)
I've done some machine learning in the past. I get how you tackle machine learning problems and the first thing that I would say that your proposed topic is not currently possible.

    "*Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news"
 (topical, I know).*"
Current AI cannot do this. This would take finding sources, pulling data out of those sources, cross referencing multiple sources, and recurring for those articles to a certain depth.

That's not a good facsimile for deciding sentencing. Sentencing is more like a linear regression classification. You have a history of previous cases where the defendant was found guilty. You then have a pile of factors that played into the judge's decision for sentencing. For example:

    * If they meant to do it
    * If they feel bad about doing it
    * If they did do it (Beyond a reasonable doubt)
    * If they have done it before
    * What severity this crime is 
    * ... etc
The judge then uses their experience in law and previous case law as well as statues to find a proper punishment. This is in the form of:

    * Time served
    * Fines
    * Privileges revoked
    * Community Service
This would then be fed into a classification engine. You leave all of the existing infrastructure in place (Judge, Jury, Lawers) and just use their decision as input into the sentencing.

Deciding the validity of claims is not within the scope of modern day machine learning (as of 2017). Classification engines are very much in the scope of machine learning of today.

I don't see how case factors could be biased. I don't see how historical cases (when stripped of all identifying information) could be biased. I don't see why a system like this would be bad.

All treatment of everyone would converge into a uniform handling of cases.

Re:Justice.exe: Bias in Algorithmic sentencing by randomtask on Sunday May 07, 2017 @09:08PM (#14288599)
> That's not a good facsimile for deciding sentencing

Sure. I'm not saying my example is the smartest (although there are people trying to use ML to do this to be fair - http://www.fakenewschallenge.org/) and I was unaware of your experience with ML, so I was working under the assumption you had no experience with it and trying to pick a simple example (even if it's dumb) to explain how bias can sneak into models in general, rather than in the specific case of sentencing criminals.

Let's loop back to what you originally said:

> If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program

You then go on to describe a number of factors that you think should go into sentencing models that leave plenty of scope for bias:

* "If they meant to do it" - this is a judgement made by a person and clearly reflects the view of the person making the decision.

* If they feel bad about doing it - again, someone has to judge whether someone is legitimately remorseful or is trying to pretend they are to get themselves a lighter sentence.

* "If they have done it before" - This will reflect things like policing tactics. For instance poorer areas might be subject to higher rates of policing especially in areas adopting the broken windows theory of policing (https://en.wikipedia.org/wiki/Broken_windows_theory#New_York...) and often in these areas petty crimes are cracked down on more frequently. This means that people are more likely to have run ins with the law, meaning they're less likely to get jobs due to convictions showing up in background checks, which in turn increases their likelihood to reoffend.

* What severity this crime is - I'm not sure what you mean by this. Do you mean e.g. murder being more severe than petty theft, or things like how severe an assault was? I'm assuming the latter since the former is often just covered by things like sentencing guidelines anyway. If someone commits an assault, then how do you rate this in a way that a model can understand? How do you ensure consistency across different cases and judges?

At any rate, the point of these models is usually to remove the biases that judges might have about people of certain backgrounds from sentencing guidelines and produce a score that informs the likelihood of the convict reoffending (I believe), so your proposal isn't how this works in practice. In practice they're trying to avoid exactly these kinds of subjective assessments you proposed and replace them with supposedly objective predictors for the likelihood of the person in question to reoffend. From the article linked from the post: https://www.propublica.org/article/machine-bias-risk-assessm...

> Northpointe’s software is among the most widely used assessment tools in the country. The company does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. (On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary.)

> Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. The survey asks defendants such things as: “Was one of your parents ever sent to jail or prison?” “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights while at school?” The questionnaire also asks people to agree or disagree with statements such as “A hungry person has a right to steal” and “If people make me angry or lose my temper, I can be dangerous.”

Given that independent research seems to confirm that the company's model seems to favour higher sentences for people of colour, it's pretty clear from that description where biases could sneak in to the model, I hope?

Re:Justice.exe: Bias in Algorithmic sentencing by hans_dampf on Sunday May 07, 2017 @10:08PM (#14288797)
The "Machine Bias" report that ProPublica published about the Northpointe software (and that is cited in OP) has been shown to be wrong.

Source: Flores, Bechtel, Lowencamp; Federal Probation Journal, September 2016, "False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks.”", URL http://www.uscourts.gov/statistics-reports/publications/fede...

In fact the ProPublica analysis was so poorly done that the authors of the above study wrote in the conclusion:

> "It is noteworthy that the ProPublica code of ethics advises investigative journalists that "when in doubt, ask" numerous times. We feel that Larson et al.'s (2016) omissions and mistakes could have been avoided had they just asked. Perhaps they might have even asked...a criminologist? We certainly respect the mission of ProPublica, which is to "practice and promote investigative journalism in the public interest." However, we also feel that the journalists at ProPublica strayed from their own code of ethics in that they did not present the facts accurately, their presentation of the existing literature was incomplete, and they failed to "ask." While we aren’t inferring that they had an agenda in writing their story, we believe that they are better equipped to report the research news, rather than attempt to make the research news."

What I find remarkable is that in the ongoing coverage ProPublica has published on this subject in December 2016 they interviewed a bunch of more people, but none of the folks that have criticized their analysis (published in September). Make of that what you will.

Re:Justice.exe: Bias in Algorithmic sentencing by randomtask on Monday May 08, 2017 @06:07AM (#14290406)
I cannot believe that the paragraph you cite actually appears in a journal article. It's really rather unprofessional and silly.

I don't want to end up defending the methods ProPublica have used as I am certainly not qualified to do that and have no skin in this game anyway. I posted here initially in response to a very general question about bias in models, and I'd rather not be drawn into a lengthy discussion about this specific piece.

However, I do have one or two issues with the conclusions you seem to be drawing in your comment:

> What I find remarkable is that in the ongoing coverage ProPublica has published on this subject in December 2016 they interviewed a bunch of more people, but none of the folks that have criticized their analysis (published in September). Make of that what you will

I'm not sure it's possible to conclude anything from that actually. There could be plenty of non-nefarious reasons for the omission. For instance, one individual they cite in the follow up review [1] has written a paper citing the paper you linked to showing that "the differences in false positive and false negative rates cited as evidence of racial bias in the ProPublica article are a direct consequence of applying an instrument that is free from predictive bias to a population in which recidivism prevalence differs across groups". [2] The Flores et al paper seems to claim that showing that predictive bias does not exist is enough, which it would seem is not the case. If racial bias might appear anyway in the situations in which the model is often applied in reality, then perhaps the ProPublica authors felt that the paper cited below [2] adequately addressed the criticism of the paper you cited and decided not to reference the FPJ article for reasons of clarity in their follow up? I think discounting their work because of a single omission would be throwing the baby out with the bathwater.

The ProPublica authors cite plenty of other research in the area in their follow ups. Sure, this is all largely in agreement with their conclusions or go further, but does this matter unless those publications are incorrect? The ProPublica authors are writing for a news publication not an academic journal and are therefore not obligated to cite every relevant publication when they're publishing. So long as they can do this without forcing a conclusion then I don't see the problem. Perhaps they deliberately ignored the paper. Who knows?

[1] https://www.propublica.org/article/bias-in-criminal-risk-sco...

[2] https://arxiv.org/abs/1610.07524

Re:Justice.exe: Bias in Algorithmic sentencing by theemathas on Sunday May 07, 2017 @06:15PM (#14288016)
Well... why can't the dataset be biased?
Re:Justice.exe: Bias in Algorithmic sentencing by amitdeshwar on Sunday May 07, 2017 @06:39PM (#14288123)
That's not a bias of the programmer
Re:Justice.exe: Bias in Algorithmic sentencing by chongli on Sunday May 07, 2017 @07:15PM (#14288226)
Unless the programmer chose the dataset.
Re:Justice.exe: Bias in Algorithmic sentencing by dogecoinbase on Sunday May 07, 2017 @06:27PM (#14288070)
How was the dataset produced?