The problem is in some sense, the opposite... our timing is now so good that we can notice the natural variations in Earth's rotation to astonishing accuracy. Thus, if we want to keep the clock in sync with what the Earth is actually doing, we need to add the leap second.
(There's quite a bit of debate about that "if", and IMHO right now industry after industry seems to be experiencing major, expensive disruptions and repeatedly incurring risk, to save astronomers from having to consult a lookup table that they of course would build into computer programs and pretty much forget about ever after. But I digress.)
The Earth's rotation really does wobble by some amount due to natural shifts in density, temperature changes, and any number of other things. It's why we don't have a schedule of leap seconds for the future. If it were merely a matter of accurate timing we'd have a schedule, because our timing is accurate enough by orders of magnitude to make a schedule for a hypothetical perfectly-steady or perfectly-slowing Earth for millennia in advance. The problem is that we can't predict what the Earth is going to do.
And simply put, the leap seconds are added so that clock noon, solar noon don't drift too far apart.
In the meantime, the Paris-based International Earth Rotation and Reference Systems Service will keep track of the gradual slowing of the planet, caused in part by drag created by the Moon. Millions of years ago days were 22 hours long.
Consider the leap year for a second. We add it because our years have 365 days is slightly incorrect (by about .25 days a year). It isn't that the number of rotations per revolution is or isn't changing. That isn't relevant to our use of a 1 day correction every 4 years. If some force caused us to revolve around the sun slightly faster and it took 365 exact, then we could remove the leap year.
Removing the leap year would be a result of our revolution being slightly faster. But the leap year itself is not.
In short, the delta of the leap correction is a result of the change in the system being monitored, but the leap correction itself is because our system of measurement is off.
Seconds are not defined by the rotation of the earth, any more than they're defined by its revolution.
Having a scheduled addition of the leap second only corrects an issue resulting from our timing being slightly off (by about 1 second every couple decades). Now, if the second had to be add in changing frequency... the changing in frequency would be a result of the earth's rotation changing... but even still that wouldn't be why the second itself is being added, is it?