News for hackers. Stuff that matters.
Re:It’s time to leave the leap second in the past by ncmncm on Monday July 25, 2022 @03:56PM (#32230098)
Yes. Leap seconds are just a thoroughly bad idea, from any angle.

Yet, not nearly so bad as Google's approach: no leap, but a 24-hour smear, during which 24 hours they are out of sync with literally everybody else in the world.

Re:It’s time to leave the leap second in the past by joerichey on Monday July 25, 2022 @04:11PM (#32230276)
For future leap seconds, Google (including GCP) are planning to use a "standard" smear (https://developers.google.com/time/smear). This is also the same smear used by AWS.

It seems like if the ITU decides to keep the leap second (a bad idea, in my opinion), the large infrastructure providers will just use the same standard smear for their clocks.

Re:It’s time to leave the leap second in the past by ncmncm on Monday July 25, 2022 @05:28PM (#32231173)
They will be wrong all day long, not just for one second. AWS will be, too, evidently.

Few others will do anything so idiotic.

Re:It’s time to leave the leap second in the past by guipsp on Monday July 25, 2022 @05:44PM (#32231337)
They will be wrong by at most one second. And being wrong doesn't even matter, as long as everyone (or rather, all your servers) agree.
Re:It’s time to leave the leap second in the past by fanf2 on Monday July 25, 2022 @06:25PM (#32231757)
At most half a second, in the middle of the smear at midnight when the leap second is applied.

The Facebook smear is asymmetrical so it starts off 1 second off just after the leap second, and subsequently corrects itself.

[ The reason Google and Amazon use a linear smear is because NTP clients try to measure the rate difference between their local clock and the reference clocks; if that is different every time the NTP client queries its servers, it will have trouble locking on and accurately matching the smear curve. You can mitigate this somewhat by fixing a higher NTP query frequency, but that’s a heavy-handed fix for an engineering mistake. ]

Re:It’s time to leave the leap second in the past by lifeisstillgood on Tuesday July 26, 2022 @02:59AM (#32235230)
They are "wrong" technically but they won't be "wrong" more than a packet getting routed transatlantic would be - so if I send two packets from my laptop in London - one to AWS in London during the smear and one to another laptop in NYC, and timestamped the packets arrivals, the time stamps would likely be similar. Yes "wrong" but if it's a problem then it's a problem you have with speed of light. The answer is to find a different / reliable method of ordering.
Re:It’s time to leave the leap second in the past by staticassertion on Monday July 25, 2022 @06:04PM (#32231531)
What are the practical issues with potentially being wrong for one day by at most one second?
Re:It’s time to leave the leap second in the past by a9h74j on Monday July 25, 2022 @06:13PM (#32231631)
I assume finance and control systems to start. I might be helpful to have a fallback time-ordering algorithm not dependent upon one monotonic clock, but then you might have a rarely-used fallback to have bugs in, I imagine.
Re:It’s time to leave the leap second in the past by Too on Tuesday July 26, 2022 @02:02AM (#32234830)
What are the practical issues for finance and control systems?
Re:It’s time to leave the leap second in the past by denton-scratch on Tuesday July 26, 2022 @05:51AM (#32236255)
For finance, it matters what order events occur in. I've never messed with control systems, but I suppose similar issues arise.
Re:It’s time to leave the leap second in the past by Too on Tuesday July 26, 2022 @08:12AM (#32237211)
Control systems, or any form of embedded safety critical system, use monotonic clocks where calendar time is a complete non-issue.

Ordering of events, on what scale are we talking here? If it’s just within a transactional database there are multitude of ways to do it. Even distributed dbs have such features without relying on perfect time. If you are looking at a spanner style db you need a lot more guarantees than “I just used the time my cloud provider assigned to my vm”, plus being in sync only matters within your own cluster?

Re:It’s time to leave the leap second in the past by GoblinSlayer on Tuesday July 26, 2022 @07:45AM (#32236978)
Ordering can be solved by a mutex.
Re:It’s time to leave the leap second in the past by a9h74j on Tuesday July 26, 2022 @05:02PM (#32243964)
I was thinking of more distributed control systems, particularly where testing for edge cases might be difficult and rare, and rigorous methods (did Lamport solve distributed mutex?) are probably off the radar in terms of culture.
Re:It’s time to leave the leap second in the past by GoblinSlayer on Wednesday July 27, 2022 @09:25AM (#32250116)
Any forum is a distributed system of a server and clients that agree with each other on order of user posts.
Re:It’s time to leave the leap second in the past by Thorrez on Monday July 25, 2022 @08:51PM (#32232970)
Meta also smears, as mentioned in the article. The article calls smearing "common practice".

I think smearing can help reduce outages. The article mentions that reddit and Cloudlare had outages caused by leap seconds when they weren't smearing. I think it's a tradeoff between being standard and avoiding outages.

Disclosure, I work at Google.

Re:It’s time to leave the leap second in the past by oittaa on Monday July 25, 2022 @05:08PM (#32230944)
AWS does exactly the same.

> The AWS Management Console and backend systems will NOT implement the leap second. Instead, we will spread the one extra second over a 24-hour period surrounding the leap second by making each second slightly longer.

https://aws.amazon.com/blogs/aws/look-before-you-leap-the-co...

Re:It’s time to leave the leap second in the past by ncmncm on Monday July 25, 2022 @05:29PM (#32231178)
I.e., be out of sync all day long with literally everybody except Google.
Re:It’s time to leave the leap second in the past by nvr219 on Monday July 25, 2022 @06:27PM (#32231772)
No, you don't understand, Amazon and Google will be right, and everyone else will be out of sync with THEM.
Re:It’s time to leave the leap second in the past by Gigachad on Monday July 25, 2022 @07:32PM (#32232352)
What is the consequence of being a fraction of a second out of sync?
Re:It’s time to leave the leap second in the past by nemothekid on Tuesday July 26, 2022 @01:04AM (#32234450)
I’m really struggling to understand the problem - what does out of sync here mean - especially when you are talking about a max 500ms. ntp clock drift can already be as bad as 500ms and if you need something tighter, then you may have already gone the gps route. Either way I can’t imagine anyone building a system that has less than a 500ms tolerance for any arbitrary computer on the internet. At the very least special relativity is going to be one the first road blocks you face in trying to get perfect universal time sync
Re:It’s time to leave the leap second in the past by remus on Tuesday July 26, 2022 @08:08AM (#32237169)
> I’m really struggling to understand the problem - what does out of sync here mean - especially when you are talking about a max 500ms.

From a purely academic perspective, it means from noon preceding a leap second to noon after the leap second there's a difference between UTC and what google (and anyone else who does smears) reports as the time. If you really hated yourself and were trying to use timestamps from conflicting sources to order transactions in a database, for example, this could mess up the order of your transactions which could have all sorts of fun consequences.

Re:It’s time to leave the leap second in the past by londons_explore on Monday July 25, 2022 @08:24PM (#32232761)
Algorithms that involve 'i did this thing at this time, everyone else should wait for that time to have passed before continuing to ensure correct ordering' (ie. Many database systems) will dramatically slow down.

Systems that use atomic clocks might have been able to do 10,000 transactions per second will suddenly only be able to do 2 transactions per second if they contain a node that doesn't use the same smearing.

Re:It’s time to leave the leap second in the past by p_l on Tuesday July 26, 2022 @03:27AM (#32235390)
Those algorithms depend on their reference time being common across all processing nodes - in fact the only system I know that uses wall clock time for inter node ordering is.... Google Spanner, which runs just fine with their smearing approach.
Re:It’s time to leave the leap second in the past by bryan_w on Tuesday July 26, 2022 @01:33AM (#32234644)
But _all_ of Google's servers will be off by the same offset, which means that they can continue to use their distributed time-based locking mechanism. Generally, if you require that high resolution of time, you don't leave your infra.
Re:It’s time to leave the leap second in the past by zeroimpl on Monday July 25, 2022 @10:20PM (#32233514)
Seems a weird way to write an algorithm. Easier to rely on database locking and/or the obvious fact that if you have received some data telling you about an event having happened, then it has happened… Curious if there are any real reports where this caused issues?
Re:It’s time to leave the leap second in the past by rrss on Tuesday July 26, 2022 @09:42AM (#32238147)
Refer to “Practical uses of synchronized clocks in distributed systems” for some info on algorithms like this.

https://dl.acm.org/doi/10.1145/112600.112601

Google spanner uses stuff like this with a clock called TrueTime. I don’t know how leap seconds / smearing is handled in that system.

Re:It’s time to leave the leap second in the past by ncmncm on Monday July 25, 2022 @11:08PM (#32233816)
If you have not heard of problems with time in distributed systems everywhere, maybe read up instead of guessing? There are literally thousands of papers on the topic, more every year. It is not considered solved.
Re:It’s time to leave the leap second in the past by nemothekid on Tuesday July 26, 2022 @01:07AM (#32234466)
If you have a distributed system that requires strict time synchronization why do you care if Google’s time is wrong? Surely if you need that level of synchronization you have your own implementation with gps clocks.

If you don’t, then you already have to deal with standard drift with ntp anyways so Googles time smear is the least of your problems

Re:It’s time to leave the leap second in the past by ncmncm on Thursday July 28, 2022 @12:04AM (#32260372)
One does not always have the luxury of systems operated entirely within one organization.
Re:It’s time to leave the leap second in the past by nemothekid on Thursday July 28, 2022 @11:30AM (#32265684)
If you don't have the luxury of managing all your systems within one organization, then time smear is the least of your problems. Day-to-day ntp drift alone will be a bigger problem.
Re:It’s time to leave the leap second in the past by zeroimpl on Tuesday July 26, 2022 @12:17AM (#32234188)
But where are the reports/blog posts from the thousands of companies using these algorithms that ran into performance problems the last few times cloud providers used time smearing? I’ve done some searching and can’t find any, which suggests this is mostly a theoretical issue.

Yes I could come up with an algorithm where this is a problem, but in most cases I’d favor a solution that doesn’t depend on separate systems having clocks perfectly in sync.

Re:It’s time to leave the leap second in the past by zmgsabst on Tuesday July 26, 2022 @12:24AM (#32234224)
Okay, how many suggest relying on a perfectly synchronized clock versus how many suggest designs which account for slightly desynchronized clocks?
Re:It’s time to leave the leap second in the past by zsims on Monday July 25, 2022 @05:53PM (#32231435)
At that point, how much of the internet is that?
Re:It’s time to leave the leap second in the past by ncmncm on Monday July 25, 2022 @06:42PM (#32231916)
Almost everybody.

AWS and Google would like to think the internet is just them, but they are a tiny little bit of it.

Re:It’s time to leave the leap second in the past by mattnewton on Monday July 25, 2022 @04:29PM (#32230489)
Didn’t Google change their smear to 24 hours because it was what AWS and some other internet time servers used?
Re:It’s time to leave the leap second in the past by ncmncm on Monday July 25, 2022 @06:43PM (#32231927)
Yes: which shows that the skew was a problem.

Eliminate UTC leap seconds, and the overwhelmingly worse smear problem vanishes too.