The "real math" comes in when you're debugging a huge network that occasionally emits NaNs deep in the reverse pass. At that point, you need to understand the details of gradient descent and a whole lot more practical analytic theory.