There isn't a huge problem with changing it, there are people who want to do exactly that and do something else instead. The prominent alternatives that I know of still do something to keep time tied to the mean solar day, though, and changing time standards can be a complicated process so it's likely to take a while.

https://en.wikipedia.org/wiki/Leap_second#International_prop...