"Changing any of these will make the summer drift into the winter, or the night into the day, or whatever."

Leap days, sure.

Leap seconds - as the article points out, there have only been 27 so far since 1970. Less than a minute per century isn't much drift, and could be corrected in a bigger block later.