There's nothing absolute about either the math or the metaphor. Both get good answers in relatively limited domains.
One obvious problem is that reality may use a completely different set of mechanisms. Physics is really pattern recognition of our interpretation of our experience of those mechanisms. It's not a description of reality at all. It can't be.
And if our system of metaphors is incomplete - quite likely, because our experiences are limited physically and intellectually - we won't be able to progress past those limits in our imagination.
We'll experience exactly what we're experiencing now - gaps between different areas of knowledge where the metaphors are contradictory and fail to connect.
If you experience pushing this object that feels to weigh 1kg with a force that feels like 1 N, you are going to experience seeing it accelerate at 1m/s^2.
But philosophy is not knowledge, and it is in fact math that is the only form our knowledge can have in this area, whether we like it or not.