>And I have always wonder if the time spent proving Gauss Markov theorem is better spent somewhere else.

No.. learning how to prove it will serve you well in the future. All deep learning is is figuring out different ways to combine all of this prior art inside of a computation graph.

Step N is linear regression, SVD etc... step N+1 is deep learning.

I wish I knew what it was like to have the math background, and don't at all mean to suggest one should not bother, but as a person on a team of people who have made a few production NN models over the past year: it is completely possible to conceptually and practically grasp iterated reverse differentiation with convolutions well enough to use deep learning to do novel work, without having the deep background math knowledge. For example, I barely know what a Support Vector Machine is, nor could I do a linear regression without a lot of hand holding. But I can design a passable tensorflow model and improve it.

It would definitely be very hard to do any meaningful research from this position, but I know enough to be useful (er, or dangerous) and can read papers and code to keep up with recent advancements, and try things out like different convolution designs, layer designs, gates, functions, feedback, etc.

(On the topic of reading: Karpathy[1] and Colah[2]'s posts have a wealth of introductory conceptual information and images in them, and helped our team discussions a lot while we've learned.

1 - https://karpathy.github.io/

2 - https://colah.github.io/)

I think it will vary for person to person - but when I am looking at a new idea(i.e. principle component analysis) that is built on top of old ideas(PCA is based on Singular value decomposition) - familiarity with old ideas makes me more comfortable/confident in thinking that I'll be able to understand the new idea thoroughly and hopefully I'll also see the incoming pitfalls of the new idea. And I really enjoy the process of adding new concept/idea in an existing larger picture.

Sure makes sense. There are multiple layers to deep learning:

The math and proofs. The algorithms. The software tools/frameworks Building NNs with those tools.

All of those have different levels of conceptualization and utility.

As some extreme examples: Anyone can build an iPhone image classifier drag and drop today. That’s one level. Developing an alternate to backprop is another level.

Another example would be databases.. there's the theory, the tools and the applications.

I liked "Hands on Machine Learning" by A. Geron. It's quite practical - you won't find careful progressive proofs, but it is very good at introducing the methods of ML and relating them to real problems. It provides you with a way to get your hands round the techniques - with an MSc in Stats you should then be well placed to understand where the gaps and problems are and to use things wisely.

It isn't quite state of the art, and it is very light in terms of how to evaluate and deliver the solutions. But I thought it was good. My team at work has gone through it selecting one chapter at a time this year in our book club - it's provided a good catch up and share forum for us. Some of the folks in the team are Engineers and so haven't a strong ML background, a couple are Data Scientists and therefore were in the same place as you (+/- some use of deep learning in a few engagements) and a couple of the folks are diehard ML researchers, but focused on different things (time series, NLP, evaluation) so there was a lot of crossover for everyone.

Last time I took a Uni. course, on the side of work, I enrolled in a brand new Deep Learning course. We used Ian Goodfellows book (https://www.deeplearningbook.org/), as well as supplementary research papers.

There is no theory in current state of the art deep learning. It works, but no one knows why, and no one knows how to fix a model that breaks.

Sounds like you might like this book: https://www.deeplearningbook.org/

SVMs are still relevant and well worth learning if you want ML of the non-neural variety.