Even though we do not have a complete understanding of exactly how the networks work, what is the function that they are minimizing but what we do know is that it is a mathematical function i.e. it has been mathematically modeled. So, I think it is safe to say that it does turn it into a game of mathematics.

Yes the function themselves are pure mathematical functions, but the way to derive them relies on human intuition (not mathematical formula). It's like saying: I know how to do a 1 times 3, I just do 1 + 1 + 1, but I can't tell you why, then I can't say that 1 times 3 is just mathematics.