> Let's say that you go to a bank and ask for a loan. The bank has a procedure to decide if they give you the loan or not. If your request is rejected, they can (hopefully) explain why you didn't qualify. Instead, the whole process gets offloaded to a learning AI that makes that decision for the bank. For the most part, we don't really understand why systems like neural networks make a decision. Now if the banks denies your loan, the best they can say why is 'huh, the computer said so'.

This is similar to the current situation in the UK for various financial and insurance applications. The application is farmed out to an organisation that ensures compliance with whatever regulations apply, but often simply gives a yes or no answer, without details. Or at least, the details are not passed back to the customer.

> What if suspiciously the banks start rejecting applications from an specific minority? Has the AI determined that this particular minority is trustworthy? Does it even matter?

This is a very real concern. Specific subsets of society tend to build and train AIs and may consider their outputs accurate or as expected, whereas other subsets of society may disagree but may be underrepresented among AI engineers. There is a danger that an AIs ability to distinguish subsets of society will be paired with value judgements. It doesn't have to be this way, we need to involve all members of society in discussions that are currently somewhat restricted.