Is it? Says who? This is an untirely unfalsifiable assumption. I personally don't share it. "It from bit"[0], I say, mathematics is fundamental, physical reality is just our perception of the underlying math.

[0] Title of a paper by John Wheeler. Can't find it right now, but if you google "it from bit" you'll find a lot of commentary on it.

The concept of fundamentality doesn't exist in mathematics, and it's by design. What is fundamental, Tailor series or Fourier transform? Mathematics doesn't care, it simply has no such concept.

Found commentary on "it from bit". It looks like an attempt to resurrect antirealism (participatory universe). The reason why antirealism was buried is because there's too much realism in quantum physics, so it won't fly.

This is exactly the position I'm trying to argue against.

What does it mean for something to be a physical structure? That you perceive it to be physical? But aren't your perceptions part of the universe? So how would you know if it actually "exists" anywhere?

Physical structure works by its own rules, mathematical structures work by our rules. For example, in mathematics you can complete an infinite process, because there's no time in mathematics, it can just be assumed out of existence. It can't be done with physics, because physics isn't affected by our assumptions.

Time and physics are just things you experience. They're parts of the universe, they're within the universe. The mathematical universe itself can quite easily be infinite in both space and time, you just can't experience all of it.