* Private, for-profit company
Either one of those determining a person's sentence is a potential for danger. The combination is a nightmare.
They aren't secret algorithms, and both the algorithms themselves and the data being fed into them are subject to review and challenge by the people (and their counsel) subjected to them.
Sure, opaque decisions made by software are creepy. But mostly, they faithfully reproduce the decisions humans (judges and legislators) were already making, and are in fact trained on those decisions. Algorithms are an easy no-guilt target to write about, but I worry that it obscures deeper sentencing issues.
In the case of this software, though, there is no cross examination. Say the secret sauce is simply a data set and Bayesian reasoning. How do we know if the data set is representative? If it is just an opaque box, that is unacceptable.
As far as I can tell, pretty much all of arson science was fundamentally broken for at least 30 years. What were the basic principles of the field - that arson fires burned hotter than others, that puddle marks indicated accelerants, that multiple-origin burns indicate arson - are all wrong.
Multiple people were executed for arson homicides on this evidence. Some were executed or even convicted after these principles were rejected. Nor was this a "science marches on" thing. The assumptions had simply been made based on observation and intuition, with no formal testing ever.
So this is the sort of thing I mean. It's not a poor expert who can be cross-examined. It's an entire field producing entirely broken results for decades using unquestioned best practices. You can't cross-examine "are even textbooks in your field a complete pack of lies?" and win.
I understand the concern, I'm not comfortable with this, but I'm also not convinced that black box software is actually much different from what we have now. The grounds for conviction and sentencing are often completely opaque to the defense already.
http://www.newyorker.com/magazine/2009/09/07/trial-by-fire
https://books.google.com/books?id=GSJ7Ja95oegC&pg=PA382&lpg=...
California's rule for `good behavior == half the prison sentence` [0], like any algorithm implemented in the real world, is ostensibly based on data analysis (studies of recidivism) and cost trade-offs. So is the 3-strikes-law [1], which vastly simplifies the cost and complexity of sentencing judgments, while making moral tradeoffs.
And like in software systems, sometimes judicial algorithms have severe bugs. The one that recently comes to mind is how sex offenders face penalties beyond prison because of alleged recidivism worries. Apparently, the belief that sex offenders are at a greater danger of recidivism was based off of an unsupported statistic found in a magazine that ended up being cited in a pivotal Supreme Court decision [2]
[0] http://www.latimes.com/local/crime/la-me-ff-early-release-20...
[1] https://en.wikipedia.org/wiki/Three-strikes_law
[2] https://www.nytimes.com/2017/03/06/us/politics/supreme-court...