He apparently invented it [1], or at least popularized it via Laplace transforms for solving differential equations.

[1]: http://hsm.stackexchange.com/questions/1984/why-is-the-heavi...

The step function is about techniques for analyzing discontinuous functions. You can create discontinuous functions by multiplying a normal continuous function by the unit step function. Which was at the time a new thing. It took the mathematical community a long time to get comfortable the techniques. Indeed if you take college math they call it the 'unit step function' in order to avoid referring to it as the Heaviside equation, which is what it's called if you take engineering math.

Constants, reciprocals, parabolas have been known since the time of the ancient Greeks or earlier. Classically they are called Conic Sections. There are other types functions that are named after people. Taylor Series and Bessel Functions come to mind (after Brook Taylor and Friedrich Bessel). There are more of course.

Because it's more fitting to give his name to a function that has a "heavy side".