I’ve wondered that before. The question would be how to differentiate the two possibilities. I suppose if you could somehow prove that our universe is the only one capable of existing while still meeting certain consistency requirements, then it might be fair to call that a “mathematical structure”.

On the other hand, if our universe is one of many arbitrary possible universes, I’d say it’s not a mathematical structure (although it could still be the only universe that exists, hypothetically).