News for hackers. Stuff that matters.
Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by mtgx on Wednesday October 26, 2016 @10:34AM (#12796724)
I'm always amused by people who see these headlines like "AI does X with 80% accuracy" and think that's high.

That's terrible accuracy. Imagine this algorithm would then be refitted to replace the judge (or augment judges in a way that judges mostly rely on the tool to make a decision, which is already starting to happen [1]) and automatically put people in prison - with only 80% accuracy. Does that number still seem high now?

[1] - https://www.propublica.org/article/machine-bias-risk-assessm...

Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by arcanus on Wednesday October 26, 2016 @11:03AM (#12796979)
I generally agree with your skepticism of \% as a meaningful metric for AI accuracy or technological prowess.

However, do we have any reason to believe that a collection of random, qualified judges would be able to predict the outcome of a court verdict with 80% accuracy? I would be surprised if judicial decisions are not stochastic.

Also, we should consider a substitution effect. According to a cursory internet search, 'As of 2012, the pay for federal district judges was $174,000 per year'. If this system is substantially less expensive to purchase and maintain, then there is an argument to be made that it is more efficient, and provides better value, even with a perpetually reduced accuracy.

Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by tpm on Wednesday October 26, 2016 @11:29AM (#12797252)
Even if it was cheaper to purchase and maintain, it wouldn't be more efficient, because money spent is not necessarily the measure of the efficiency of the justice system. That could be, for example, lower criminality. If a system fails to jail enough criminals to do that, even if it's cheap, it's not working. And also, false positives (e.g. jailed innocent people) are especially troubling and can be very expensive in a justice system.
Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by notahacker on Wednesday October 26, 2016 @11:25AM (#12797199)
I suspect a collection of random, unqualified members of the public could predict the outcome with 100% accuracy using the written judgement as the basis for their evaluation, as appears to be the case in this study...
Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by dragonwriter on Wednesday October 26, 2016 @05:06PM (#12800554)
> However, do we have any reason to believe that a collection of random, qualified judges would be able to predict the outcome of a court verdict with 80% accuracy?

I think a random selection of second-year law students could probably beat 80% (by about 19 percentage points) predicting the outcome of a court decision from the facts and law sections of the same decision.

Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by rayiner on Wednesday October 26, 2016 @03:11PM (#12799488)
Most cases are quite routine and easily predictable.
Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by josaka on Wednesday October 26, 2016 @10:58AM (#12796935)
It's tough to evaluate the usefulness of the 80% claim without a baseline. If a simple heuristic of "plaintiff wins" is correct 85% of the time, the model isn't that impressive.

In the long run, such a model would likely be confounded by the parties, who'd model and adapt their arguments to drive the model to a favorable result or settle cases where the model provides a strong signal to which they can't adapt. (Which is not to suggest that this wouldn't be a very socially valuable result.)

Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by CapitalistCartr on Wednesday October 26, 2016 @10:47AM (#12796846)
They only get better, never worse. And the use is not for actual judging, but to estimate different approaches to a case. Tactical analysis, basically. Both sides plus the media have a serious use for this, so money will flow.
Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by logfromblammo on Wednesday October 26, 2016 @11:20AM (#12797150)
This will certainly be used by professional legal counsel to help determine whether they should take your case, and how best to charge you for their services.

Cases predicted to win might be taken on contingency, for a fraction of the judgment award. Cases predicted to lose might be pay-as-you-go for each appearance or filing.

Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by falcolas on Wednesday October 26, 2016 @11:48AM (#12797448)
I think it's the 80/20 problem. I've heard of a lot of algorithms which hit the 80% mark, but getting hire than that is too cost ineffective. So ineffective that it's usually cheaper to hire low skill workers to validate the output of the algorithms, especially when accuracy matters.

I've heard of it at Google and Bing for popular searches. I've seen it firsthand at a company applying classifiers to twitter posts. Microsoft took how long to go from "80%" voice recognition to their stated 96% (with generous assertions about the environment and language)? Apple/Google/et.al. still rely on humans to act as validators for the training of thir personal assistants.

I have trouble seeing for-profit companies ever truly getting higher than the 80%; it's just too expensive when cheap labor also exists.

Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by dragonwriter on Wednesday October 26, 2016 @05:02PM (#12800527)
It doesn't really make sense to talk about replacing the judge (judges, really, since this is specific to the European Court of Human Rights) with this AI, anyway, since this AI "predicts" based on portions of the document written by the Court to explain the decision it has made. It's like replacing a playright with an AI that predicts Act III from the content of Acts I and II.
Re:AI ‘judge’ can predict court verdicts with 79 per cent ac... by empath75 on Wednesday October 26, 2016 @12:09PM (#12797634)
> I'm always amused by people who see these headlines like "AI does X with 80% accuracy" and think that's high.

"High" is a relative term. At one point, a chess program beating any human was an impressive feat. Today, any human beating a chess program is an impressive feat.