It's an interesting way to view these things for sure. Marvin Minsky believed that it doesn't make sense to talk about the physical reality of our world, he thought it was a possible world and that's all that matters.

I take my cue from the physics of information theory and computation. In that view a computation is a process physically performed by a system. The same applies to mathematical calculations. The result exist independently in some transcendent sense, but in order to actually obtain a result a physical system has to perform the process. That physical system exists independently of the abstract mathematics, and independently of any other system performing the same calculation, or computation.

In the same way my brain can have a thought. I can perceive a colour. The pattern of activity might be extremely close to the activity associated with you perceiving the same colour. It might be identical with the pattern of an identical clone or twin of mine experiencing the same perception. Nevertheless they are separate, unconnected, completely independent activities that just happen to have similar interpretations.

That's a strict physicalist view, and it's one I subscribe to as being the most plausible. Nevertheless I'm a human being, I have children, I care about my family and friends, and humanity generally. So clearly patterns matter to me as a person, but that's just in my nature. I don't see any reason to think it's fundamental to anything, because thinking that way IMHO leads to some very wooly thinking. just because gold is shiny and pretty, that doesn't mean I think there's something cosmically magical or special about those wavelengths of light. The experience and appreciation of things is important to us, but it doesn't define reality.