a) it ensures that they're positive so they don't just cancel each other out and b) like you mentioned, it penalizes huge errors more heavily.

There are also historical reasons for using squared error. The square function is smooth and differentiable to you can analytically solve for the gradient. Before fast computers this was crucial for solving regression problems as a closed for makes everything easier.