Or more succinctly: mathematics doesn't give us truths, it gives us consequences. The statement, "2 + 2 = 4" is not an empirical fact intrinsic to the universe. It's a consequence of our definition of natural numbers. It's considered a useful definition because (among other things) of its application as a foundation for engineering and the sciences. But many other useful areas of mathematics have essentially no basis in reality.
Then physics isn't a science either since only parts of the field fits this criteria. I think you will realize that the distinction between physics and math isn't as clear cut as you imagine, it is not uncommon for physicists to publish in math journals and vice versa.
Ultimately though, the math that is natural science is just called “physics”, right? ;) The reason math can’t be a hard science is because we can’t prove abstract math by physical experiment. That’s all “hard science” means - things you can show by physical experiment. We can prove math theorems by showing they’re consistent under all the other math theorems, but generally not by conducting physical tests.
I think turning math into a purely abstract game came later, and we owe our fun to the discipline's more humble origins.
There are of course other viewpoints that math has a Platonic metaphysical reality in and of itself, independent of the natural world.