Offering them a clear separating criteria in the passing and non-passing portions of the space of all properties considered is definitely a good idea. And, as an explanation / description, it's independent in form of the algorithm used.

Well you don't train the algorithm to just give you Yes/No. You tell it to give you

a) Yes, SUPER green

b) Yes, Green

c) mmmmmmmmmmmmmmmmyeah, ok, but watch this guy

d) please review manually

e) no, too many other loans

f) no, but maybe yes, with more collateral

g) no, convicted felon

h) no, unemployed

i) no, your wife has a billion loans

j) no, insufficient money down

k) ...

(there's like 300 or so different reasons in practice)

So the inputs to the neural net would be all the information you know about the applicant, appropriately encoded. Then you sprinkle hidden layers to personal taste, and the output layer would be a->i binary neurons.

This seems to miss the whole point of ML: you want to consider all the features holistically. As your GP said, you could tweak one parameter until it passes. But which parameter? If you have 300 features about an application, but they have an income of 10 billion dollars, of course they're accepted. If they have 0 income, they're rejected. You don't need a fancy algorithm to reject people for the obvious reasons you've described. The problem is the fuzzy gray area where I make 100K a year, but I seem to have too much debt. You don't want to kick this back to a human and say 'do it manually', you want to say 'Based on all his features, his risk of default is x%. This is (higher|lower) than our threshold'.

edit: Just to clarify, for banks it doesn't make sense (in my mind) to train a classifier based on a bank manager's past decisions. I would train a regression based on the historical default rates, so the algorithm would predict the probability of the client defaulting on the loan.

No you don't. As the parent poster says, if you screw up, you want to be able to figure out why. Unless you enjoy digging for months for biases in large incomplete datasets with a panicked guy screaming at you, you'll have at least a couple dozen output neurons (you can always also have a yes/no one if you like).

In practice all the parameters will fire to some extent, right. You need some function to resolve it. Generally people start out by giving the "no's" preference, then you switch to something more complex, then some product manager suddenly declares the law: give "yes" preference.

So the output of that network will be a) 15.15% b) 74.3% c) 55% d) ... and so on. The highest values are considered and some function chooses from that.

It's how it works in the human brain as well, but the solution that wins is the one that gets control of the brainstem. Generally being the first to respond is pretty much guaranteed to get you control, but something else may override you later. The human mind, in reality, is not a single whole, but is ~300 separate minds that could, if it were necessary, operate alone. Each of those is constantly coming up with responses to inputs, and they're all linked through the brainstem, which is also the link to the rest of the body (except the eyes, which are only input to ~6 of them, and the ears, which also are connected directly to ~6 of them. That seems to be to enable brain-eyes bandwidth to be ~100000 times more than all the bandwidth it has to the body, and brain-ear bandwidth ~100 more).

True, but I believe the argument still stands for other ML algorithms, such as support vector machines. There is no way to query directly why the SVM maps an instance to a label.

It seems like in SVM the best way to find out 'how to pass' is to find the shortest path from your point in the feature space to the separating hyperplane. The length of that vector in each axis explains how much you need to change to cross the threshold.

How about you run 20 vector machines ? Each only identifies how likely one reason is (using distance from separating plane as a "how sure is this SVM" measure). Besides, SVM do not have a particularly good reputation due to the need for good kernels, which requires either massive computing power (just have 10 million kernels) or people with very good domain knowledge. And if you need people with very good domain knowledge, why bother with AI ?

AI is hard and people who know both a lot about statistics, AI and programming AND a lot about, say insurance claims, well good look finding them. People with good knowledge about insurance claims and good knowledge of excel, they're, well, not a dime a dozen, but certainly much easier to find.