I agree with you, except that math does have a relationship with the physical world.

Math was invented and refined over time in order to find ways to model, communicate and reason about our observed physical phenomenon.

I understand though, that there is a ton of subject matter within the field that is completely divorced from the "real" world.