There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example.

Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable.

[0] https://christophm.github.io/interpretable-ml-book/neural-ne...

Indeed ML is vast and not limited to NN.

My question was more towards people who are actively using NN as a solution and _have_already_ shipped products in a 'regulated' industry.

The products are obviously out there, so to me it is confusing what evidence can a team provide at this point in time (with the current NN implementations) in order to get a product using NN (with a non-deterministic uncertainty) approved by a regulatory body? (automotive, medical, etc.)

Examples that jump to mind are Tesla's stereovision sub-systems (I am assuming NN are used), or NN based medical imaging classifier software (NN instead of Clustering or SVM techniques)

Here is a (longish) post I wrote about this issue (mainly from the perspective of verifying autonomous vehicles):

https://blog.foretellix.com/2017/07/06/where-machine-learnin...

Thank you very much on point. Thanks for sharing!

DARPA has acknowledged this problem and is working on it.[1]

See slide 11 for some of the technical approaches to making AI more interpretable.

[1]https://www.darpa.mil/attachments/XAIProgramUpdate.pdf

Various model compression schemes get rid of high-frequency noise in the model, combating over-fitting. This results in robustness against adversarial inputs.

Some of these schemes are motivated by rather robust theory. Gaussian Processes are a good starting point to learn more.

This is certainly out of my depth but how would this model 'reduction' improve provability?

It would (I assume) reduce instances of non-desirable behaviour. But how would it improve the evidence I am able to provide to a regulating authority?

To my layman's understanding; by reducing instances of undefined behavior as those instances should come from the noise of the data. Noise looks like structure in very-high dimensional views of data, so you need bounds on that stuff.

For autonomous vehicles the 'proof' to the regulator can be the signature of the engineer. - Fortunately some engineers have higher standards. - I think in the future the trainig data will be from simulations that create curriculum learning datasets so that the noise characteristics are perfectly known. The ML algorithms can be written with dependent types, so that you can prove your code does what you think it does.

Another challenge is inductive bias, which is a lot like confirmation bias. This bias comes from choosing an ML algo which is sensitive to certain information and blind to others. You need to navigate the set of all possible functions, AKA Hilbert space, to overcome it. Fortunately only a small corner of this space is relevant to our universe and Tensor Networks seem to address this problem.

It looks like a lot of work to put these pieces together but at least it looks like the problem is tractable.