Physical structure works by its own rules, mathematical structures work by our rules. For example, in mathematics you can complete an infinite process, because there's no time in mathematics, it can just be assumed out of existence. It can't be done with physics, because physics isn't affected by our assumptions.