Currently our calendar goes off by one day in 3236 years. If the history is calendars is indicative, in about 10,000 more years we may change our calendar. (Or, being multi-planetary by then, maybe we'll consider it a quaint relic of our origin.)
Our clocks predict astronomical time of day to less than a minute for a human lifetime. And both DST and timezones demonstrate that we're happy to live with the clock and Sun disagreeing by an hour or more.
My position is that both are currently good enough for the next couple of thousand years. And we can let our distant descendants sort it out when the time comes.
It’s more likely that every planet will have its own calendar and you’d have to do the conversions and translations. That’s what scientists do with Mars time. Thanks to relativity, time at point A is not the same as time at point B, and whether they stay in sync for you depends on how you get from A to B.
While "now" can vary according to the time it takes to get from A to B, all observers can work out what "now is according to the reference frame of the distant stars". On a mere interplanetary scale, this idea is amazingly precise. We might not agree on how much time passed (this has already mattered for GPS satellites), but for all practical purposes "now" is perfectly meaningful.
If you want to use real-world time, you have to make sure it stays in sync with the real world. Switching to something which is close-but-not-quite-correct will cause even more issues than we are currently having with leap seconds. Can't deal with that? Well, just use the Unix timestamp like the rest of us?