I think the real issue is that tech journalists and laymen don’t know enough to understand. You can’t explain multivariable calculus to someone that doesn’t know algebra. Or option 2/3 are the problem.
I'm sure I'm not alone in the fear of living in a (not so far) future dystopia where our driver's licences get cancelled because of our driving behavior caught on traffic cameras. "What did I do that caused it to get cancelled?", "We don't know, better luck next time."
"Your test got flagged for cheating, it's a very sophisticated system with IR vision, audio detection and grade assessment, there's nothing I can do."
"Our theft-preventing camera system says you are suspicious, you're gonna have to go now."
When all we have to respond is "its the algorithm", I don't think much people will be ok with that. Youtube for instance has gone down this rabbit hole and a lot of content creators and users are less than happy to receive that answer.
I’d say these errors are not due to the inscrutable nature of ML but bad communication.
There are various issues with this desire, including the wrinkle that humans are not always good at explaining their own decisions and we persistently assume we are. But that's neither here nor there. People like feeling in control, and ML tends to expose that they're not.
Also why do we call them "hidden" layers instead of "intermediate" layers? They're no more hidden than any other data in memory, are they?
the first neural network-like things would have input nodes directly connected to output nodes. Thus both layers were 'exposed' and visible. The hiddden layers are 'hidden' because you don't directly touch them or use them-they are obscured. You can call them intermediate layers if you want I suppose-people will probably understand what you mean
2) The authors fully understand the tools they're using, but feel there's an element of unexplainable meaning in them that I wager isn't actually there.
3) I'm ignorant and I think this is explainable when I just don't know enough to know it's not.
I've done machine learning projects, and if asked to explain to a "non ML trained" person, I'll try like this:
You know how if you get a scatter plot, you can draw a line through the middle of the points and make predictions off the y=mx+b function of that line?
Well, that's what we're doing here, except the function isn't a line, it's a shape with as many dimensions as we have features.
Let's pretend the only features we've got are lat and long. Imagine you asked for us to create a classifier to predict the depth at a given lat and long over the grand canyon. Each neuron is a function, I think of each neuron like it's casting a "line" through a scatter plot along its particular axis. Then we combine all the lines together and weave a mesh. The act of training simply adjusts the details of all the simple functions, trying to seek out the sweet spot in that point cloud, so our mesh is close enough to the grand canyon to be useful. Too many neurons, and the mesh wont be flexible enough to make useful predictions from. It'll be overfit!
Now, we might be able to get decent performance out of a model built out of x,y->z -- but some thoughtfulness and domain knowledge TENDS to make models perform better. Lots of the "unexplainable magic" comes from these hunches being tested and providing benefits to accuracy. I'd wager without seeing any data that if we add two new features: normal direction of the rock face and speed of the Colorado river at that point -- we can make the model much better for negligibly additional training time.
The more compute time and data we've got, the more features we can throw into the model to see if they help the usefulness of the mesh.
I've got a metaphor to explain why some features drop out during training that involves gimbal lock here if people are still asking questions. It requires non "trained" folks to have seen Apollo 13. Fortunately no non-technical stakeholder ever cares enough to make it there though. :D Shipping a product before the sun burns out makes this a decent time to introduce PCA/SVD or whatever dimensionality reduction techniques I used this time around.
If the solution ends up being a fever-dream abortion of an LSTM/Perceptron/Markov chain that I came up with while sleep deprived after a heavy night drinking -- I just say I came up with the model in an ad hoc manner, and the cross validated R^2 scores speak for themselves.