On the one hand, opaque bureaucratic processes already exist today. It's not like the bank that refuses you a loan will tell you "oh, but we would totally have granted it if you were 20% richer and also not a woman, so come back after you've achieved that and we'll give you the loan"!
On the other hand, even opaque models can be studied and systematized. You can't run a simulation of a judge on 100 sample cases, then on the same 100 sample cases with names changed to sound like immigrants, and numerically measure the judge's bias. You can do that with a ML model, and compensate in various ways.
That's because they don't want to tell you, not because they don't know. The bank knows exactly why they refused to give you a loan and exactly which series of policy changes led to that outcome. They also know exactly how to change the policy to achieve a different outcome.
None of those things are true with AIs.
That sounds like a hellish dystopia.