It is relatively clear why it works beyond the training and validation set: What is being approximated is a smooth function, which in the case of a classification task is a function from the space of things to be classified (images of a certain size) to the n-simplex, where n is the number of classes. Then the preimage theorem tells you that over a regular point of this smooth map lies a codimension n submanifold in the space of things to be classified. That in turn can be interpreted as the submanifold of all things that look like the class you are assigning, especially close to the corners of the n-simplex (being a regular point is an open condition). In short: Because the map is constructed to be smooth it will make sense beyond whatever the training / validation data was. Note that this does not guarantee that it has learned something reasonable about the dataset, just that it will have found some way to smoothly separate it into different components.

The question is why dont kernel methods (with appropriate smoothness inducing kernels) and relatedly Gaussian processes do as well. I am being deliberately presumptuous, it is not a proven result that RKHS based methods would be dominated by DNNs. These two approches, RKHS and DNNs arent that fundamentally different, and we have known that even before the recent explosion of interest in DNNs. That infinite DNNs converge to Gaussian processes is an old result.

Of course there are differences, once you choose a kernel the feature map is set in stone, on the otherhand DNNs can search over the space of feature maps. It is likely that the function that a DNN converged to was already in the Hilbert space associated with the kernel, but that does not mean one would converge to it in a finite amount of training data.

Once one goes to infinite dimensions all separable Hilbert spaces are the same (up to isomorphisms) but with finite data it matters what basis / kernel one chooses.

Some days I think I'm smart, then I read stuff like this. I wish there was an EILI5 Plugin for chrome that I could use to tag any paragraph and someone would come along and break it down for me. I'd do it for your comment!

Maybe the ELI5 plugin could be reasonably approximated by adding links to all phrases with corresponding Wikipedia articles, like:

https://en.wikipedia.org/wiki/Validation_set

https://en.wikipedia.org/wiki/Smooth_function

https://en.wikipedia.org/wiki/N-simplex

https://en.wikipedia.org/wiki/Preimage_theorem

https://en.wikipedia.org/wiki/Codimension

https://en.wikipedia.org/wiki/Submanifold

I guess it depends on whether you're willing to read a textbook's worth of Wikipedia articles. (There is no shortcut. Actual 5-year-olds also require a few years until they can understand advanced mathematics.)

You might be confusing "smart" with "knowledgeable". There's no way to be smart about concepts you haven't yet learned. And the ELI5 idea is cool - just need to find a business model :)

Yes, but these things should be in the guide, or at least with lower-dimensional framing to make it understandable.

General popular opinion seems to be that these are (to greater or lesser extent) a mystery for everyone. Can you suggest any intermediate reading on things like generalisation? I've looked online but only found either "here's how to recognize numbers in the MNIST dataset using numpy" or "First we take <long string of squiggles> which trivially implies <longer string of squiggles>..."

That is because hyperparameters are indeed a mystery and the fine tuning of those is more an heuristics based art, than a science.

The Approximation Power of Neural Networks https://towardsdatascience.com/the-approximation-power-of-ne...