To state that "the physical world came first" is to assert the primacy of existence in opposition to the primacy of consciousness (e.g. Descartes) -- that's correct and that's metaphysics.
I want to close with the observation that whether you're aware of it or not, you're talking about supernatural claims that have been postulated without any scientific evidence.
To make this distinction between the words metaphysical and supernatural feels a bit similar to fundamentalist Christians who assert that they're not religious - a phenomenon which I suspect happens for the same reasons: because they believe their faith represents the "truth" about the universe, it doesn't count as a religion. But to everyone else, it still does.
I subscribe to the evidence of the senses, logic, and science. If you think any discourse about such topics belongs to the supernatural and religious, that's your problem, not mine.
Physics as a science does not depend on metaphysics in any way. When a physicist does his/her work, it does not require the invocation of supernatural principles. While speculation and hypotheses are essential to the scientific process because they drive further exploration, metaphysical arguments on the other hand are just postulates about the nature of reality - and they are in most cases either in direct conflict with observational data, or designed to be un-disprovable.
Now for physics, as in the actual universe. This doesn't depend on metaphysics either. In fact, it's the other way around. As the universe develops, intelligent life emerges, humans in our case. These humans can then invent religion and the supernatural. The physical world came first.