You don't need math 'the language', but only because your ad-hoc inferences are suitable. You do need math 'the models' because you can't think without those things. The subject of math isn't actually esoteric. It's the community and conventions that are esoteric.