Something not very intuitive is there's no ab initio (i.e. "blank slate", a priori, purely theoretical, etc.) notion of "legitimacy" in mathematics, or even physics. There was a lot of debate on whether complex numbers were "legitimate". The truth is, they work well in several physical models; that's where their legitimacy comes from. We assume that any crazy axiomatic system can model reality, well, almost any actually. There are some properties excepted of any axiomatic system, like self-consistency, and of course the ability to generate some kind of complete (or locally complete at some domain) model of reality. What we then do is try to come up with models and infer the correct or most useful ones from experiments.

This means if a physical model yields a sum like 1+2+3+..., we of course expect the result to be unphysical and the model invalid. However, if experimental data is consistent with a result like -1/12, that would be a clue to explore a different arithmetical system for our model, one which generalizes summations (to Ramanujam summation or other kinds of sums).

In mathematics the situation is even more interesting: the legitimacy is driven by taste of researchers, which will blend physical and philosophical relevance, and even aesthetics, curiosity and interest in part of the mathematician. (This is why we understand mathematics to be almost art, although a very particular art of course)