Well, then it sounds like your reasoning gets it backwards: the axioms that produce systems without significant consequences or connections outside of their own abstract realm end up being ignored.

Or in other words: the constraints on maths are imposed from outside of maths.