In any case, ideally the framework should touch on geometry eventually as well, but I am not aware of concrete ways to do it. Toposes are considered geometric, but I don't know whether geometric in the sense of GR. The eventual or at least, hopeful, end point is something like [2] where your mathematical framework should ideally in the language of categorical or functorial duality express physical dualities. I don't know how to get to GR, as I say, and should read maybe this [3] but there are many ways to get to QM. One example is through entropy: the generalised abstract set subobjects are partitions and partitions encode entropy through "distinctions", i.e, elements in separate equivalence classes of the partition. The category of sets itself is a model for QM theories via it's inspiration for (symmetric) monoidal categories so perhaps one would want to express in the self-dual language that as well. It's one thing to point to multiple uses of category theory in physics, but it is another to keep track of it in your personal, internal intuition strategy.
[1] https://arxiv.org/abs/1808.01350
Pure theoretical physicist might be in a comparable budget, but I think it is true that the lower expectations on math research is partly due to how cheap it is to fund.
But for someone focused on mathematics, string theory, quantum loop gravity (if that is what it's called) and all the others become tools and objects of study on their own that have a unique perspective to offer in their own right. Whether you study one or the other then becomes a personality inspired preference. Just because I don't study number theory, doesn't by default scorn the field (or perhaps I just can't do number theory).