As an example, just last week a (huge) paper [1] was put on arXiv that used these theoretical methods to analyze a bunch of common architecture building blocks (skip connections, normalization, etc), and then applied their theoretical findings to figure out how to train Resnet like models in similar training time without these seemingly "required" building blocks.
Deep Learning is still in its infancy in many ways, and this type of research takes time, slowly building on successive results.
team behind the paper is Deepmind/Google. It is probably worth a read.
The same statement could have been said of neural networks for a decades, but researchers poking at corners eventually found methods to turn them into the useful tools they are now. After all, those methods you downplay were created since neural networks were not useful at that time, whereas many of these were.
I'd not poo poo what researchers decide to poke at. Pretty much every breakthrough is people poking at the edges of understanding. If solutions or steps were straightforward, then it would be engineering, not research.
>My prediction is that t
Mine is that neural networks as we understand them now get replaced by much more solid methods, based on the principles from scientific machine learning, where sophisticated differentiable models that are designed to mimic the problem space get tuned. After all, even current neural networks are heading that direction. Neural networks are simply too simplistic to capture lots of the complexity that problems demand (hence the current move past them in many domains).
Gluing linear functions together ad-hoc is simply a low level approximation to what can be developed using centuries of powerful mathematics to make models.
Maybe so. But I suspect that scientific machine learning would be difficult to grasp by most working data scientists and ML users who don't have graduate-level training in a quantitative science or engineering, which implies an uphill battle for adoption. You can do a lot with ML and deep learning without to construct a sophisticated mathematical model of your problem space.
(I took one class in differential equations in 2004, touched up on that in a boot camp before starting a PhD in econ, and have probably forgotten everything I ever learned about the topic ever since. Why yes, I feel mathematically inadequate sometimes.)
Most of what those same people do now was difficult to grasp when they started. The next group will adopt whatever is the best tools for the problems they face.
>which implies an uphill battle for adoption
The same was true for current ML and data science. It was true for OOP. It was true for structured programming. This is always true, but tools that solve problems get adopted.
>You can do a lot with ML and deep learning without to construct a sophisticated mathematical model of your problem space.
True, but as such simple tasks are mined out, everyone doing much work will move past that. For example, pick any task on say paperswithcode.com and look at the top performing models. It's hard to find any that are simple old school networks. All of them I just sampled do much more sophisticated modeling of the precise problem space, and the trend is that all current edge and future work will involve modeling the problem much more carefully.
I expect the naive deep learning approach will be a tiny blip in the progress of using learning to solve problems.
Future progress looks like it relies more and more on making a good model of the problem with some untuned parameters, then using gradient descent or related to tune those parameters. The better the original model, the better it performs, and the less data one needs to train it since it has relevant structure already.
For example, learning basic physics in a net takes significantly more training, parameters, and cost than simply building in basic physics and tuning the small unknown parts. The net version also doesn't scale well since it is built from pieces (linear chunks) that simply don't match functional relations in the underlying problem.
This is like criticizing the physicists working on the principles underlying steam engines during the days when people were making empirical advances in building steam engines, but nobody had figured out all the core principles yet.
Of course they understand that it’s an empirical science today. That’s exactly why they’re doing the work they’re doing.
I respectfully but completely disagree.
All the enabling technologies of deep learning come from machine learning and statistical learning theory. Stochastic gradient descent, regularization, dimension reduction, bootstrap, bagging, boosting: these techniques remain the fundamental tools in the deep learning toolbox, and a constant source of inspiration for the latest innovations.
Physics has done next to nothing for deep learning in comparison. It's as marginal as the kernel SVM stuff. Just fiddling around with the same old stat mech / network theory / power law stuff the Complex Systems types been doing for the last few decades.
Stat mech works great for materials because materials are relatively simple things and we have relatively simple questions about them. We want to know how they respond to electricity, magnetism, heat, pressure, etc. When you turn to elaborate gadgets for machine translation or image recognition, sure, you can ask similar questions, but the questions just aren't as interesting. You'll get plenty of plots and histograms and power laws out of it, but it's all going to be very superficial. It's not going to tell you how the gadget works.
All that said, there's no way to be sure where the big advances in deep learning theory are going to come from. Your guess might be as good as mine.
I think some researchers are refusing to accept the idea that machine learning is very much an experimental science today, and the (very cool) mathematics of kernels, SVMs, empirical risk minimization, bayesian statistics, etc. are simply no longer useful in the large scale regime.
My prediction is that there will be a useful "deep learning theory" in the future, but it will look a lot more like physics (such as Kaplan's scaling laws) than early 21st century machine learning mathematics/statistics.