It's not actually hard to come up with a valid and understandable explanation from a neural net, or from any other black box decision maker as long as the decision maker is consistent.

Let's take simplified case, to make visualization easier. Suppose the black box is deciding whether to approve mortgages, and it only has three inputs: income, credit score, and mortgage amount.

You can visualize the inputs as points in a 3-dimensional space, each point representing a unique combination of income, credit score, and mortgage amount.

Now imagine in that space we place green dots at all the points that represent inputs that result in approval of the mortgage application. Do nothing at the points for inputs that represent rejected inputs.

Need to give someone an explanation of their rejection? Find their point in that space, and then find points on the green region that are nearest their point, and tell them what would need to change to move their point to those green points.

I'd suggest, in this 3-dimensional space, to look at four green region points. (1) the nearest green point reached moving from their point parallel to the income axis, (2) same but using credit score axis, (3) same but using mortgage amount axis, and (4) the nearest green point.