I wish I knew what it was like to have the math background, and don't at all mean to suggest one should not bother, but as a person on a team of people who have made a few production NN models over the past year: it is completely possible to conceptually and practically grasp iterated reverse differentiation with convolutions well enough to use deep learning to do novel work, without having the deep background math knowledge. For example, I barely know what a Support Vector Machine is, nor could I do a linear regression without a lot of hand holding. But I can design a passable tensorflow model and improve it.

It would definitely be very hard to do any meaningful research from this position, but I know enough to be useful (er, or dangerous) and can read papers and code to keep up with recent advancements, and try things out like different convolution designs, layer designs, gates, functions, feedback, etc.

(On the topic of reading: Karpathy[1] and Colah[2]'s posts have a wealth of introductory conceptual information and images in them, and helped our team discussions a lot while we've learned.

1 - https://karpathy.github.io/

2 - https://colah.github.io/)

I think it will vary for person to person - but when I am looking at a new idea(i.e. principle component analysis) that is built on top of old ideas(PCA is based on Singular value decomposition) - familiarity with old ideas makes me more comfortable/confident in thinking that I'll be able to understand the new idea thoroughly and hopefully I'll also see the incoming pitfalls of the new idea. And I really enjoy the process of adding new concept/idea in an existing larger picture.

Sure makes sense. There are multiple layers to deep learning:

The math and proofs. The algorithms. The software tools/frameworks Building NNs with those tools.

All of those have different levels of conceptualization and utility.

As some extreme examples: Anyone can build an iPhone image classifier drag and drop today. That’s one level. Developing an alternate to backprop is another level.

Another example would be databases.. there's the theory, the tools and the applications.