Chomsky wants linguistics to be mathematics.
We know it doesn't work this way once we actually start building a real model and move beyond what is really just exploratory data science. We'd be nowhere with physical science if the mental models we construct (which have nothing particularly numerical about them in the first instance) did not actually have a coherence that is both amazing and extremely productive.
This is not some Platonic drift or remotely philosophical. This is how physics is done. Even casual familiarity with the history of physics and reading the writings of key thinkers would point to this.
Heck, in quantum mechanics the relation between the physical/mathematical model people came up with the actual "measurement" is profoundly non-trivial and does not remotely fit your bizarre reductionist "programme".