https://www.chrisstucchio.com/blog/2015/no_free_samples.html https://www.chrisstucchio.com/blog/2015/ab_testing_segments_... https://www.chrisstucchio.com/pubs/slides/crunch_conf_2015/s...
Now, most of the folks pushing bad stats are simply not understanding the subtleties of a complex field. PP, in contrast, did understand and ignored the results they got. Yep, that's dishonesty in my book. What would you call it?
I have no idea what you are talking about with "machine learning" - it's a synonym for "statistical analysis", leaning a bit in practice towards external validation of models without regard to internal validity. Which is, incidentally, what PP was trying to do.
If you want to draw some fine line where a random forest is machine learning but a Cox model isn't, be my guest. I'm sure that's a great reason to ignore PP's own analysis.
And you can't stand that they went out and talked to people...
No, I can't stand that they cherry-picked anecdotes to push a conclusion that's the exact opposite of what their own analysis claims.
Now if you want to criticize the analysis, go ahead. But that's not really a good way to defend PP - after all, if the analysis is bunk, doesn't that discredit the whole article?
um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics
If you want to make that argument, do it. But that's not the argument PP made.
As for my own biases, I'm pretty open and honest in my beliefs in the power of technology. I also back up my claims with math that actually agrees with me, unlike PP. Do you have any objection to what I wrote beyond the fact that you dislike the tone, and might define "machine learning" slightly differently than me?
You have no idea what machine learning is? Then why do you use that phrase. My interpretation is that it is differentiated from statistical analysis in that there's a system that adapts its heuristics to trained and incoming data. For someone to call COMPAS "machine learning" is to have a faith that this system would be adequately tuned and adapted to changing data. For example, if the factors that predicted higher recidivism for blacks were first devised in an era where low-level drug possession was a criminal offense...and now that era no longer exists, and there's no mechanism to adjust for that...I would say that would be a big fucking problem for every criminal who is processed under COMPAS today (regardless of the argument that drug possession crimes were more punitive towards minorities).
But we don't know because COMPAS is secret, which was the main thrust of this article. And it seems to perpetuate the biases of the "non-machine" justice system, except in an untransparent way.
I said:
> um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics
you said:
> If you want to make that argument, do it. But that's not the argument PP made.
Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be.
> Scores like this — known as risk assessments — are increasingly common in courtrooms across the nation. They are used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts — as is the case in Fort Lauderdale — to even more fundamental decisions about defendants’ freedom. In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing.
I agree with you that 0.05 < 0.057. What we disagree on is how much of ProPublica's article is based on that, and whether that difference is enough to call them bald-faced liars, given that they acknowledge the limitations of their data collection and show their work and their data. They're attempting to reverse-engineer a black box that's being paid for by taxpayer dollars. They shouldn't have had to provide an anecdote-laced story with statistical analysis to pressure the judicial system for more transparency, but that's what they've chosen to do.
The Cox model fits that definition.
Perhaps you mean ML is synonymous with reinforcement learning, which is I think the closest thing to "tuned and adapted to changing data"?
My interpretation of ML is that it's about evaluating statistical predictors without worrying about their internals. Those are the fundamental advances that have given us all the classical "machine learning" algorithms - random forests, SVMs, neural networks, etc.
And the entire point of those techniques is that you can use them to evaluate a secret algorithm.
Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be.
The entire article claims the algorithm is both inaccurate and racially biased, which their statistical analysis directly contradicts.
They never once made the argument that an accurate and unbiased model should not be used.
But the fact that you keep referring to COMPAS as "machine learning" when neither COMPAS (in the various papers I've read on their site) nor the ProPublica article describes it as such makes me wonder if you're missing a few key things about the actual reality. AFAIK, COMPAS clearly involves statistical heuristics, but its not clear the data pipeline used to create those heuristics, nevermind how it would feed into a machine learning system. And that's kind of the problem here, with so much of this algorithm being proprietary, there's no way to keep it accountable, other than the imperfect ad-hoc way that ProPublica devised.
You have a highly emotional reaction to their process -- calling them flat-out liars -- despite the fact that they were the ones who not only procured the COMPAS scores via FOIA, but the also wrote their own scraper to automate the collection of criminal records from the Broward court system. They then reconciled this with the databases from Florida corrections and jail records from Broward's sheriff's office, and the databases required manual reconciliation. They then packaged and posted all of this data for public use, along with their workbook, on to Github.
And this is the work, in your opinion, of a group of deliberately dishonest journalists who are looking to sell clickbait. And you can't stand that they went out and talked to people (anecdotes, as if data weren't the plural of anecdotes...and as if COMPAS itself isn't built on anecdotes because it requires practitioners to go out and interview inmates in person, for fucks sake)
Again, I haven't run through the numbers myself, so I can't agree or disagree whether ProPublica has conclusively argued that race is a statistically significant factor when COMPAS fucks up the prediction. Or if the "bias" that they speak of is more focused on the "hazard ratios" differentiation between racial groups, and perhaps that's not as sexy a bias as its headline warrants.
But I'll be honest. Your conflation of COMPAS with "machine learning", and your simplistic view of the law and the justice system -- "we've taken a flawed and racist human system and replaced it with a much better machine learning system"...um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics -- makes me think that you come into this fight with at least as raging a bias as that of the ProPublica journalists.