The change in rotation is not consistent. Earthquakes change the earth's rotational inertia by raising and lowering land near the equator. Sometimes they jog the entire planet forwards or backwards in its rotation by a slight increment. Tidal interactions vary significantly enough to have an effect. Even man made changes are measurable - the three gorges dam in China moved enough water uphill to have a measurable effect.

And simply put, the leap seconds are added so that clock noon, solar noon don't drift too far apart.