The “look for symmetries and conservation” edict is a direct emanation of Emily Noether's (First) Theorem, which basically places conserved quantities of a system of differential equations with the symmetry groups that system must exhibit. A deep, amazing, beautiful, bi-directional relationship between differential dynamics and abstract group algebra.

Yes, but not in a way that's germane here. Physics in the sense of the workings of the world is fundamental. Physics in the sense of science practiced by humans is infamous for spherical cows, point masses and mathematical funny business. Not the same thing at all. :)

That's a discussion that doesn't make much sense.

What do you mean by "more fundamental"? The fact is physics and mathematics treat entirely different subjects. Physics treats the physical, empirical world. Mathematics is the study of formal thought. It isn't meaningful to compare them that way.

Theoretical physics "shut up and calculate" is experimental mathematics.

Or perhaps at the root they are the same thing.