Absolutely !
For squared loss almost every theorem you want to be to be true are actually true. In a way its dangerous because it sets up bad expectations (* no nerd pun intended).
A result that holds for squared loss but not for others is the SVD as a low rank approximation of a matrix.
*Expectation minimizes the squared loss
The negative log of the (unnormalized) Gaussian density is basically x^2 and this fact comes very handy.
It's used extensively in Bayesian regression, modeling noise as additive Gaussian leads to the squared loss function, and if you also model the prior as Gaussian it leads to an L2 regularizer.
It also simplifies many other calculations. Fitting a mixture of Gaussians by expectation-maximization or the Kalman filter come to mind.