That also has a formal mathematical interpetation as a distribution rather than a function now and the theory of distributions found many applications in PDEs.

An interesting thing is that before Dirac, Green was already informally using distributions in the 1830s to solve differential equations with what we now call Green's functions.

Closely related to Green's functions are the fundamental solutions to an elliptic operator, consider the Laplacian and its fundamental solution in Rⁿ (n≥3), what's the Laplacian of the fundamental solution? A Dirac's delta!

The big trouble with the delta function in quantum physics is that it's in some sense the natural basis for your continuous Hilbert space, but they aren't in the Hilbert space. Which necessitates a whole lot of mathematical explaining-away-formalism.