Part of me believes that geometry is really just a subset of human thinking that comes very naturally/quickly to us, and thus we are more successful studying physics through geometry than through less 'algebraic' frameworks. So we made the most advances there just because we are biased toward doing so.

But then I think that we may be so strongly biased toward doing so because there's something fundamentally easy about evolving a brain that comprehends geometry. That information with geometric representations are fundamentally easier to evolve good mental models for than other kinds of information.

You've reminded me of Plato's quotes:

"God ever geometrizes."

and

"Geometry existed before the creation."