The idea that noon is not when the sun is on the meridian and a mere footstep east or west changes time by an hour with all the force of the state's instruments of violence [1] is a mark of tyrannical authority. The Gregorian Calendar is a product of a system in which people act upon a belief in a divine mandate for their authority.
Which brings me to the objection that was not addressed. I don't want to cede my freedom to a theocracy even dressed in technocratic robes. My time isn't yours to decide.
[1]: aka, don't be late for court 'cause the judge will throw your ass in jail.
I think you might just have a problem with the concept of timekeeping, and not with this particular system.
I will concede, though, that the hypothetical situation of being late for a court date is plausible near a time zone boundary. More reason to eliminate time zones, right? The time is the same everywhere, so nobody is going to make a mistake due to a time zone discrepancy.