I know the speed of rotation changes slowly. But that would be a reason for lengthening a day, not adding a leap second. The leap second is added because our concept of time is slightly off, but it keeps being presented as if we had to add an extra second because of the slowing of the rotation. If that was the case, it would be like those 2 hours that were added in the last few million years. We don't consider them leap years because they aren't a correction but a redefinition (ignore for a moment that humans weren't around to define as a day as 22 hours back then).

Consider the leap year for a second. We add it because our years have 365 days is slightly incorrect (by about .25 days a year). It isn't that the number of rotations per revolution is or isn't changing. That isn't relevant to our use of a 1 day correction every 4 years. If some force caused us to revolve around the sun slightly faster and it took 365 exact, then we could remove the leap year.

Removing the leap year would be a result of our revolution being slightly faster. But the leap year itself is not.

In short, the delta of the leap correction is a result of the change in the system being monitored, but the leap correction itself is because our system of measurement is off.