News for hackers. Stuff that matters.
Re:Training Computers to Find Future Criminals by Smerity on Tuesday July 19, 2016 @12:10AM (#12119576)
As someone who does machine learning, this absolutely terrifies me. The "capstone project" of determining someone's probability of committing a crime by their 18th birthday is beyond ridiculous. Either the author of the article hyped it to the extreme (for the love of everything that's holy, stop freaking hyping machine learning) or the statistician is stark raving mad.

The fact that he does this for free is also concerning, primarily as I doubt this has any level of auditing behind it. The only thing I agree with him on is that black box models are even worse as they have even worse audit issues. Given the complexities in making these predictions and the potentially life long impact they might have, there is such a desperately strong need for these systems to have audit guarantees. It's noted that he supposedly shares the code for his systems - if so, I'd love to see it? Is it just shared with the relevant governmental departments who likely have no ability to audit such models? Has it been audited?

Would you trust mission critical code that didn't have some level of unit testing? Some level of code review? No? Then why would you potentially destructively change someone's life based on that same level of quality?

> "[How risk scores are impacted by race] has not been analyzed yet," she said. "However, it needs to be noted that parole is very different than sentencing. The board is not determining guilt or innocence. We are looking at risk."

What? Seriously? Not analyzed? The other worrying assumption is that it isn't used in sentencing. People have a tendency to seek out and misuse information even if they're told not to. This was specifically noted in another article on the misuse of Compas, the black box system. Deciding on parole also doesn't mean you can avoid analyzing bias. If you're denying parole for specific people algorithmically, that can still be insanely destructive.

> Berk readily acknowledges this as a concern, then quickly dismisses it. Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race.

There are so many proxies for race within the feature set. It's touched on lightly in the article - location, number of arrests, etc - but it gets even more complex when you allow a sufficiently complex machine learning model access to "innocuous" features. Specific ML systems ("deep") can infer hidden variables such as race. Even location is a brilliant proxy for race as seen in redlining[1]. It does appear from his publications that they're shallow models - namely random forests, logistic regression, and boosting[2][3][4].

FOR THE LOVE OF EVERYTHING THAT'S HOLY STOP THROWING MACHINE LEARNING AT EVERYTHING. Think it through. Please. Please please please. I am a big believer that machine learning can enable wonderful things - but it could also enable a destructive feedback loop in so many systems.

Resume screening, credit card applications, parole risk classification, ... This is just the tip of the iceberg of potential misuses for machine learning.

Edit: I am literally physically feeling ill. He uses logistic regression, random forests, boosting ... standard machine learning algorithms. Fine. Okay ... but you now think the algorithms that might get you okay results on Kaggle competitions can be used to predict a child's future crimes?!?! WTF. What. The actual. ^^^^.

Anyone who even knows the hello world of machine learning would laugh at this if the person saying it wasn't literally supplying information to governmental agencies right now.

I wrote an article last week on "It's ML, not magic"[5] but I didn't think I'd need to cover this level of stupidity.

[1]: https://en.wikipedia.org/wiki/Redlining

[2]: https://books.google.com/books/about/Criminal_Justice_Foreca...

[3]: https://www.semanticscholar.org/paper/Developing-a-Practical...

[4]: https://www.semanticscholar.org/paper/Algorithmic-criminolog...

[5]: http://smerity.com/articles/2016/ml_not_magic.html

Re:Training Computers to Find Future Criminals by Smerity on Tuesday July 19, 2016 @12:47AM (#12119750)
From "Developing a Practical Forecasting Screener for Domestic Violence Incidents":

"We had data from the screening instrument for 671 households. ... For the 516 households with complete data, there was at least one return call for 109. Thus, about 21% had a return call within three months after the screener information was collected. Although there is no guarantee that all of these calls were for domestic violence incidents, no doubt most were."

Admittedly this is one paper, and not necessarily related to the system he created for parole prediction, but this would be a worryingly small dataset (and likely bad set of assumptions) for any data scientist, machine learning researcher, or machine learning practitioner.

As Francois Chollet noted on Twitter, it's very reminiscent of the random forest approach to launching drone strikes[2] that had insanely small amounts of data:

"The NSA evaluates the SKYNET program using a subset of 100,000 randomly selected people (identified by their MSIDN/MSI pairs of their mobile phones), and a a known group of seven terrorists. The NSA then trained the learning algorithm by feeding it six of the terrorists and tasking SKYNET to find the seventh."

[1]: https://www.semanticscholar.org/paper/Developing-a-Practical...

[2]: http://arstechnica.co.uk/security/2016/02/the-nsas-skynet-pr...

Re:Training Computers to Find Future Criminals by Houshalter on Tuesday July 19, 2016 @01:52AM (#12120024)
> There are so many proxies for race within the feature set.

Yeah but, so what? Surely you don't believe race is a strong predictor after controlling for all the hundred other things? Algorithms are not prejudiced and it has no reason to use racial information when so much other data is available.

Even if somehow race was a strong predictor of crime in and of itself, so what? Lets say economic status correlates with race, and it uses that as a proxy. It still isn't treating a poor white person different than a poor black person.

And if it makes a prediction like "poor people are twice as likely to commit a crime", well it's objectively true based on the data. Its not treating the group of poor people unfairly. They really are more likely to commit crime.

Re:Training Computers to Find Future Criminals by pdkl95 on Tuesday July 19, 2016 @02:16AM (#12120108)
> Surely you don't believe race is a strong predictor after controlling for all the hundred other things?

It can be if you select the right data, algorithms, and analysis method.

> Algorithms are not prejudiced

That's correct. However, the selection of algorithm and input data is heavily biased. You're acting like there is some sort of formula that is automagically available for any particular social question, with unbiased and error free input data. In reality, data is often biased and a proxy for prejudice.

> It still isn't treating a poor white person different than a poor black person.

I suggest spending a lot more time exploring how people actually use available tools. You seem aware of how humans bring biased judgment, but you are assuming that the creation of an algorithmic tool and use of that tool in practice will somehow be free of that same human bias? Adding a complex algorithm makes it easy to hide prejudice; it doesn't do much to eliminate prejudice.

> Its not treating the group of poor people unfairly.

Yes, it is. The entire point of this type of tool is to create a new way we can pre-judge someone based not on their individual behavior, but on a separate group of people that happens to share an arbitrary set of attributes and behaviors.

The problems of racism, sexism, and other types of prejudice don't go away when you target a more complicated set of people. You're still pre-judging people based on group association instead of treating them as an individual.

Re:Training Computers to Find Future Criminals by Smerity on Tuesday July 19, 2016 @02:18AM (#12120118)
Algorithms _can_ be prejudiced. See [1]. Not being defined with bias or prejudice does not make them impervious to it.

If a dataset is biased, as you have stated by "not trusting judges" elsewhere, then the algorithm will follow that. If the dataset contains a feature that is highly indicative (be it racial information or otherwise), your model may come to depend entirely on it. The presence or absence of other data may be irrelevant, especially if you don't perform regularization or another similar tactic.

Racial (or specific feature) profiling is a very complex topic. If there's open legal questions as to whether policemen are allowed to use it, I don't feel algorithms should be allowed to.

[1]: "How big data is unfair: understanding unintended sources of unfairness in data driven decision making"[1], by Moritz Hardt, a Google machine learning researcher who has published on the topic - https://medium.com/@mrtz/how-big-data-is-unfair-9aa544d739de...

Re:Training Computers to Find Future Criminals by rrmm on Tuesday July 19, 2016 @01:16AM (#12119875)
I share your concerns. The post-9/11 scramble for homeland security funding in academia really killed my enthusiasm for the AI/ML stuff I was doing at the time.

The message that is heard when tech people sell AI/ML is 'magic cure-all' and it basically comes down to not understanding Bayes' theorem: If your algorithm has any capacity to make type one or type two errors (even if the success rates seem impressive) bad things will happen when you look for infrequent events (terrorism/crime) with low priors: you get more bogus hits than true ones. Once the priors get to 50% great, but then you have bigger issues (civil war and chaos). Law enforcement is then obliged to follow up on those bogus hits which given the size of the population can be overwhelming.

And that is even before you get to ethical minefields mentioned above. Making decisions about people based on models that the implementer might not even fully understand is unnerving. It makes it easy to slip in social/socio-economic assumptions about people for those who would like to try. Even if the implementer acts in good-faith, it can easily happen inadvertently. Given how little is known about how to deal with social problems in general, trying to paper over that ignorance with AI/ML makes everything worse. It is operationalizing ignorance with high-efficiency.

Re:Training Computers to Find Future Criminals by doctorpangloss on Tuesday July 19, 2016 @01:52AM (#12120026)
> but you now think the algorithms that might get you okay results on Kaggle competitions can be used to predict a child's future crimes

Do you think anything can be used to predict crimes? Can you imagine any fair application of justice that doesn't boil down to an algorithm?

Sentencing guidelines are an algorithm. It's literally a table on page 3[0]. Do you think sentencing guidelines, which were invented to address unfairness in the justice system like this guy's work was, are unfair?

What alternative do you propose? Case-by-case judgements? Even if you could find me a model judge, be honest: the judge probably uses something that boils down to an algorithm to be just.

> brilliant proxy for race

Suppose we only used variables with a low correlation to race. Or we decorrelate them from race somehow. Suppose error on the validation set increased. Would this be sufficient to make this system more just to you? Would a reasonable person think it is more just?

> Specific ML systems ("deep") can infer hidden variables such as race

Considering how scientific you are trying to be about your understanding of ML, you're being dishonest about how much ML systems can "infer hidden variables." There's strictly anecdotal evidence of "deep"ness inferring something hidden. We think GoogLeNet is seeing higher-order features, but then we show it a picture of a cat and it thinks it's looking at a paper towel.[1] You're overstating the power of these systems to serve your point, ironically exactly what you accuse this guy of doing.

> Resume screening, credit card applications, parole risk classification... potential misuses

I can see how to an individual these things are unfair. But how about in aggregate? If I can show you a way to process credit card applications that makes more profit, reduces bankruptcies and increases credit usage, would you say, "don't use it"? So what if it's made with ML, or pixie dust? Would you reject these things?

> destructive feedback loop

That's just society dude :)

[0] http://www.ussc.gov/sites/default/files/pdf/about/overview/O... [1] https://codewords.recurse.com/issues/five/why-do-neural-netw...

Re:Training Computers to Find Future Criminals by Smerity on Tuesday July 19, 2016 @02:44AM (#12120219)
The issue is not whether justice can be codified. Let us assume that you can and it's outlined on page 3 of your document. Providing examples to an ML model don't guarantee it will codify anything similar to that. This is a fundamental issue. Most machine learning models aren't highly interpretable and can also act in unexpected ways.

Re: (a) variables with a low correlation to race / (b) dishonest about how much ML systems can "infer hidden variables":

(b) The example you point to is specifically adversarial modification to the input data. I'm not sure how that's relevant to your point at hand except to say that ML models can be misguided when given slightly modified input. It doesn't show that inferring hidden variables is anecdotal. If you look at the work of machine translation with attention, it explicitly learns how to do alignments even though it is never given training data for it. We can visualize that[1].

(a) There is interesting work on trying to produce variables with a low correlation to race. One recent example[2] has two networks use the same intermediate representation created from the raw input. One of the networks tries to identify a particular variable (race / gender / ...) while the other tries to perform the overall objective task. We aim to modify the way we generate the intermediate representation such that the former network has a harder time identifying the sub groups whilst allowing the latter group to continue to improve on the original task. This is recent however. There are many other interesting explorations.

Regarding when to use features that may leak information, this is referred to as the fairness / utility tradeoff. If you want to see an example of both that and trying to prevent information leakage about gender in a dataset on adult income, refer to "Certifying and removing disparate impact"[3].

Note: the models used by the statistician in the article do not appear to feature anywhere near this level of analysis. Ensuring to avoid features that result in strong bias or prejudice against specific groups in a dataset is still an open problem.

[1]: https://devblogs.nvidia.com/parallelforall/wp-content/upload... (image from https://devblogs.nvidia.com/parallelforall/introduction-neur...)

[2]: http://arxiv.org/abs/1511.05897

[3]: http://arxiv.org/abs/1412.3756

Re:Training Computers to Find Future Criminals by cfallin on Tuesday July 19, 2016 @01:23AM (#12119899)
Given that ML is likely going to be used more and more -- even if not for criminal justice applications, at least in the private sector (credit scores/loan applications, insurance, etc) -- I wonder if there's a way to develop standardized "anti-discrimination standards" and then apply some regulation to any ML algorithm that makes a "life-altering decision" by some definition?

E.g. -- a "differential discrimination" test of sorts: altering any one variable of a "protected class" such as race, gender, religion, etc. does not change the answer. You would maybe want to pick a set of canonical test profiles among real people who differ only on one axis (as closely as possible), rather than just take a test point and alter one axis directly, because you'd want all the relevant correlations (ZIP code vs wealth, etc) to remain authentic. The end result would be a set of "equivalence classes": sets of human profiles who must be considered equivalent on all relevant life-altering judgments.

Or perhaps a "unit test"-like approach: similar to how one creates a unit test for each bug one fixes, create a "criminal justice ML algorithm test suite" with canonical profiles and their results: you must judge this person to likely not re-offend, you must judge that person as a high risk, you must judge this person worthy of a home loan for $X, etc. Sort of like a body of case law. I guess the risk is overfitting -- so maybe this data set is held in trust by some regulatory agency and not revealed.

People have probably thought about this and I haven't read your links -- is building a test data set and building regulations around it something that's considered?

Re:Training Computers to Find Future Criminals by Smerity on Tuesday July 19, 2016 @03:14AM (#12120292)
The European Union are likely to accelerate much of this type of research. They recently introduced regulations specifically targeting algorithmic decision-making and a "right to explanation" for automated decisions.

There's a Wired article[1] on the general scene and a paper that investigates the impact it may have on the industry[2].

Regarding differential discrimination, check out these two papers[3][4] - they're very similar to your idea :)

[1]: http://www.wired.com/2016/07/artificial-intelligence-setting...

[2]: http://arxiv.org/abs/1606.08813v2

[3]: http://arxiv.org/abs/1511.05897

[4]: http://arxiv.org/abs/1412.3756