So basically what you are saying is that, we can't be so certain that we humans invented math. Math is there, structures were there to be discovered ? Why is there such a thing as monster group ? We named those things, but if we didn't exists, it would still be there in the abstract world ? This is what is bugging me lately, I can't decide if math was invented or not. Some parts may be invented, but some parts may be discovered.
Accordingly "the working philosophy of most mathematicians is a mostly unexamined Platonist-Formalist hybrid ... Math is a product of human minds but not bendable to human will. Exploring it is like exploring a new tract of country ... but the mathematical countryside does not come into existence until you explore it."
As for infinities... I'm not sure if they drove Cantor insane or if he was already insane. Some infinities are bigger than others, and I have strong opinions about infinity.
0.999... = 1
Horrible! ;-)
To say "I have three apples" is merely a way of modeling a configuration of the world. The apples may be in my pocket, or in my hand, or in a bag I'm holding, or even on my desk in front of me. They may be in the barn out back, but in all cases we can say I have 3 apples. Saying "i have 3 apples" is a simplification of a ton of configurations of the state of the world where I have ownership of apples. Similarly I have -3 apples is a summary of all the configurations of the world where I owe three apples.
Neither is any more real than the other. And yes it is philosophical.
Are symmetry groups not "physical" enough? I can rotate a piece of paper in the plane and then undo this operation just fine.
Algebraic and topological structures seem to manifest themselves readily within nature. I should think infinities provide greater concern.