However, do we have any reason to believe that a collection of random, qualified judges would be able to predict the outcome of a court verdict with 80% accuracy? I would be surprised if judicial decisions are not stochastic.
Also, we should consider a substitution effect. According to a cursory internet search, 'As of 2012, the pay for federal district judges was $174,000 per year'. If this system is substantially less expensive to purchase and maintain, then there is an argument to be made that it is more efficient, and provides better value, even with a perpetually reduced accuracy.
I think a random selection of second-year law students could probably beat 80% (by about 19 percentage points) predicting the outcome of a court decision from the facts and law sections of the same decision.
In the long run, such a model would likely be confounded by the parties, who'd model and adapt their arguments to drive the model to a favorable result or settle cases where the model provides a strong signal to which they can't adapt. (Which is not to suggest that this wouldn't be a very socially valuable result.)
Cases predicted to win might be taken on contingency, for a fraction of the judgment award. Cases predicted to lose might be pay-as-you-go for each appearance or filing.
I've heard of it at Google and Bing for popular searches. I've seen it firsthand at a company applying classifiers to twitter posts. Microsoft took how long to go from "80%" voice recognition to their stated 96% (with generous assertions about the environment and language)? Apple/Google/et.al. still rely on humans to act as validators for the training of thir personal assistants.
I have trouble seeing for-profit companies ever truly getting higher than the 80%; it's just too expensive when cheap labor also exists.
"High" is a relative term. At one point, a chess program beating any human was an impressive feat. Today, any human beating a chess program is an impressive feat.
That's terrible accuracy. Imagine this algorithm would then be refitted to replace the judge (or augment judges in a way that judges mostly rely on the tool to make a decision, which is already starting to happen [1]) and automatically put people in prison - with only 80% accuracy. Does that number still seem high now?
[1] - https://www.propublica.org/article/machine-bias-risk-assessm...