I think part of it is also that physicists and engineers are the ones primarily teaching the subject. So it might not be that they can't handle the mathematics, but that they simply prefer to think entirely in intuition land where they can spin functions around without anyone protesting.