There's plenty of math still. In order to not waste GPU-hours, you gotta have some intution as to what _not_ to do, because otherwise you'll be trying for a very long time and relying entirely on luck, which is combinatorially unfavorable and frustrating. That intuition is often grounded in math, mostly differential calculus and statistics. That said, there are a good number of techniques which nobody is sure why they work. I.e. batchnorm, which is used all the time. People have some theories and intuitions as to why it works, but nobody is sure why, nor is there any rigorous way to pick hyperparameters for it.