I started by building an intuitive understanding of advanced physics before I really dug into the math. It is absolutely possible. That being said, I did have a very good understanding of advanced mathematical concepts to start with, so I could think intuitively about things like high-dimensional spaces and eigenfunctions.