Well also the idea of physics as the field we currently have didn't exist much before the 17th century. Movement of bodies, astronomy, fluid dynamics, electromagnetics, optics, etc. all kind of were their own thing (if they existed at all). Fundamental developments in calculus in the late 1600s enabled these subjects to be collected under one method of study/analysis which we now call physics. As much of modern math follows from the lineage of calculus the border between the things being modeled and the tools for modeling them is naturally kind of blurry, however the distinction did still exist quite strongly throughout this entire period. Look at ex. probability or algebra, although often researchers were pursuing both physics and math, they were aware that the subjects were distinct.