I don't think OP is really wrong here. Wasn't there a debate in the late 19th century that basically asked if math had to have some mapping to the natural world, or should it work independently? I though there was some argument about this with Hilbert and Poincaré about this, and Hilbert more or less won.