The collective human decision to make days-per-year vary (requiring leap rules to calculate days) seems similar to the collective human decision to make days-per-month vary (requiring month names to calculate days). So if we say a "stopwatch year" suffers the same fate as the "stopwatch month" then it's a slippery slope to saying the "stopwatch minute" is no different than a "stopwatch year" (requiring leap rules to calculate seconds) even if, for all practical purposes, it seems exempt.
I guess this is why we make "second" the SI unit, and none of our human-convenience rules mess with the duration of a second. A leap second changes the duration of a minute (and above), and a leap year changes the duration of a month (and above). Which, oddly enough, demonstrates an inconsistency: we ought to say "leap day" instead of "leap year" if the thing being added will follow the word "leap."