If I hand you a 175B parameter language model, are you really contending we know what's going on in there? At a mechanical level, sure, it's tensor products and activation functions, but that's like saying we know how human brains work because the standard model is very predictive.
Note I'm not saying we perfectly understand everything, or that our understanding is as solid as that involving convex models or linear regression. But "black box" just isn't true anymore and just adding unnecessary mystique. We are somewhere between Newton and the Lord Kelvin and Faraday era of physics, no longer in the ancient alchemist days
A big part of machine learning is looking at weights and outputs to make sure the results are sane and that you have an understanding of what's going on. This is true no matter what algorithm you use to make predictions.
I mean, even the example given by the OP about the tanks is super well known (apocryphal[0]) and doesn't require math knowledge to avoid. You just have to have heard of this kind of failure mode
Yes exactly. You have to be aware of it, you have to know what it entails and what can cause it and how to diagnose and fix it.
That’s the other half of the domain knowledge, and just “autoML-ing it” or following some set of prescribed steps won’t necessarily get you that solution.
Because of that, doing deep learning consists of a bunch of cobbled together heuristics for getting good results and probing the model to give a human an intuition for whether it's learning correctly. The tricks and tips for steering that black box have mostly been developed in the last decade: it is not a super deep well.
These tricks and heuristics are like the knowledge needed to be a technician in a nuclear facility, not the knowledge needed to build the nuclear facility in the first place. It's not nothing, to be sure, but unless you're a researcher developing new novel architectures, a very shallow understanding of the statistics will go a very long way.