This is almost correct.

It's not that math is vastly larger (or more sophisticated) as a field. Both fields infinitely large in many senses. Rather, the number of respectable starting points where you can do interesting things is much larger, orders larger in math.

There are plenty of unexplored things in physics too; the key word is "respectable".

Looking from the outside, physics suffers a lot from fashion/hot trend tendencies, where you need to be doing the "hot" thing to make the jumps necessary to the coveted Tenure Track — and otherwise, you get kicked out.