As a data scientist that heavily uses DL in my day-to-day, yes you need to know math. We are having difficulty hiring, and few candidates can answer basic questions: what is an FFT, what is the central limit theorem, what is an eigen value, what does the trace of a matrix tell you, etc. Many candidates use ML but have no idea what they are applying to the data.