News for hackers. Stuff that matters.
Re:One second per second is harder than it sounds by chubot on Monday July 21, 2014 @11:47PM (#8067300)
I think the question is: when does it matter? What applications rely on an accurate clock? I guess it would matter if you are running some algorithm on logs that uses time as an input. Interested to hear any other examples.

DJB has a clockspeed package that addresses this problem. Anyone used it?

http://thedjbway.b0llix.net/clockspeed.html

It seems interesting but I haven't (yet) run into the need for super accurate clocks (or perhaps I am not running enough machines).

Re:One second per second is harder than it sounds by sophacles on Tuesday July 22, 2014 @11:11AM (#8069370)
Power grids require highly accurate, distributed clocks.

Well they don't strictly require them, but having that capability increases efficiency, robustness and safety.

They need to be accurate because electricity is fast - so a lot of protection schemes need to have a decision made across very long distances (which breaker, at which point on a 100 mile long transmission line should be fliped to isolate faults or stop "bad things" propagation) with minimal impact to the system as a whole.

They also need to be accurate because a good measure of frequency and phasor at geographically disparate points allows system tuning. A tiny change - e.g. speed up one of the 100s of generators could result in a much smoother, overall more-efficient transmission system. Similarly, adjustments to capacitor banks and transformers can provide slight phase adjustments for optimal power flow. But these resources are hundreds of miles apart - e.g the western half of the US and Canada (and parts of Mexico) (except for texas) is one big system.

Being a large, physical system, with feedback, and also being electricity, you get some very interesting effects to try and reason about. There can be very slow (relatively) oscillations in the system, but at the same time you have many, many high frequency noise signals mixed in to the elecrical signal. To find trends and patterns for the slow oscillations you need a very high sample rate to determine what noise to ignore. And these samples need to be time aligned, because even a perfect wave will have different phases at different points along it's propagation.

Anyway - that's an example I'm familiar with. I hope it made sense, seeing as I may not have had quite enough coffee yet.

Re:One second per second is harder than it sounds by keeperofdakeys on Tuesday July 22, 2014 @02:00AM (#8067563)
As soon as you have any kind of highly-distributed system, accurate clocks are usually very important. When events can start on one machine, and end on another, it's important to have some kind of consistency (knowing that one event occurred before another). Probably the most famous paper on this is Lamport's "Time, clocks, and the ordering of events in a distributed system" - http://dl.acm.org/citation.cfm?id=359563.

But to give a practical example. On something like facebook, if you receive two comments on a post within a millisecond, how do you determine the order that they appear in the list? Each request may hit a different datacentre in the world, yet somehow the machines determine the ordering, and every request sees that ordering from then on. You can't just have a single server, because that couldn't handle the load for the entire world. These kinds of things require very strict temporal ordering, which is why time is such an important thing. This leads to the CAP theorem, and distributed systems in general.

Re:One second per second is harder than it sounds by chubot on Tuesday July 22, 2014 @04:46AM (#8067902)
So that's exactly why you don't depend on hardware clocks -- you use software techniques when your algorithm depends on some notion of time.

The point of Lamport's paper is that time in distributed systems is a partial ordering, not a total ordering. If you are taking values from hardware clocks, you are using a total ordering.

Re:One second per second is harder than it sounds by tlarkworthy on Tuesday July 22, 2014 @11:29AM (#8069484)
Google spanner is a DB that uses atomic clocks distributed world wide. They seem to think the pain involved dealing with partial ordering is not worth the hassle given there genuinely is a total ordering on operations.

http://en.wikipedia.org/wiki/Spanner_(database)

Re:One second per second is harder than it sounds by dllthomas on Wednesday July 23, 2014 @01:47AM (#8073087)
"given there genuinely is a total ordering on operations."

I thought relativity says there genuinely isn't (but it's close enough we can fudge it...)

Re:One second per second is harder than it sounds by tlarkworthy on Wednesday July 23, 2014 @07:46AM (#8073812)
ha! I think its ok as long as all parties involved are travelling on the earth and not around it near the speed of light

http://en.wikipedia.org/wiki/Relativity_of_simultaneity

Re:One second per second is harder than it sounds by dllthomas on Wednesday July 23, 2014 @10:34AM (#8074562)
Right. Like I said, we can (presently) fudge it. When we get moving faster that ceases to be the case.
Re:One second per second is harder than it sounds by keeperofdakeys on Tuesday July 22, 2014 @04:57AM (#8067928)
Yeah, but you can't have your clocks drifting a lot. It's fine if they are consistently wrong, but when a second isn't a second, things can start to screw up.
Re:One second per second is harder than it sounds by judk on Tuesday July 22, 2014 @10:14AM (#8069068)
Does Lamport's design use seconds at all, or use only "happens before" statements?
Re:One second per second is harder than it sounds by chetanahuja on Wednesday July 23, 2014 @04:13AM (#8073360)
"As soon as you have any kind of highly-distributed system, accurate clocks are usually..."

Let me stop you right there. If you have a highly distributed system, it's design CAN NOT rely on perfectly synchronized clocks among it's components. That's actually the point of the Lamport paper you cited as well as the OP. I actually started writing a detailed rehash of the Lamport's paper but I realize I'd do a poor job of it at best. Instead, I think it's best to just point to excellent and highly readable entire paper here: http://research.microsoft.com/en-us/um/people/lamport/pubs/t...

Re:One second per second is harder than it sounds by Dylan16807 on Tuesday July 22, 2014 @02:15PM (#8070630)
Using a highly accurate clock to solve ordering like that is punting on the actual problem. It's possible for requests to different servers to happen at exactly the same time. You need a mechanism to handle that. And any mechanism that can handle that can also handle clocks being a couple hundred milliseconds out of sync.

At a certain level of precision you have to accept that data only travels at the speed of light and there is no way to have a global ordering that shows data consistently and immediately. So you can use an easy non-immediate method like a vector clock.

Re:One second per second is harder than it sounds by michaelt on Tuesday July 22, 2014 @05:36AM (#8068043)
Clock years out of sync: All SSL certificates appear invalid.

Clock minutes out of sync: "Here's a temporary security token, I don't think it's expired" "That credential has expired" "This never came up in testing, I think I'll crash"

Clock seconds out of sync: Log files on different servers can't be merged reliably. Different requests went to different load-balanced servers and now you can't tell what order the logged events happened in.

Re:One second per second is harder than it sounds by spydum on Tuesday July 22, 2014 @08:06AM (#8068469)
And that last one has had many people chasing ghosts for years. I can also attest to time sync killing a number of SAML assertions and their window of validation.
Re:One second per second is harder than it sounds by MichaelApproved on Tuesday July 22, 2014 @12:29AM (#8067392)
Communicating with other servers is a good example.

I need to have an accurate clock to within a few seconds when I sign Amazon S3 files. I have code that gives access to restricted files on S3. To give access, the customer makes a request from my server, I authenticate, sign a new URL and redirect them to the signed S3 URL.

For security reasons, the URL gets a signature that expires after a few seconds. The expiration date is based on X seconds from my servers clock. If my servers time differs from S3 server time then, according to S3, the link could be expired before it's even created.

In theory, a redirect takes a second or less to complete so you'd think an expiration that's 5 seconds in the future would be enough. The reality is, over time, the server clock can differ as much as 1 minute or so from S3's clock. So, I just set the expiration to be 120 seconds into the future and call it a day.

Re:One second per second is harder than it sounds by throw_away on Tuesday July 22, 2014 @12:55AM (#8067453)
If ultrashort expiry time really mattered to you and ntpd not reliably working, you could always make a request of S3 and use the response's version of the date (which presumably is the version of time that really matters, as far as S3 is concerned).
Re:One second per second is harder than it sounds by ajanuary on Tuesday July 22, 2014 @02:53AM (#8067662)
For a bunch of distributed systems nodes need to be roughly matching real time, but also within some tollerance of each other; it's simplest to make everything as accurate to real time as possible.

I work in broadcast automation. It's a distributed system and the clocks need to be within a certain tolerance of each other. Historically that's done by having a dedicated piece of timekeeping hardware that dictates the station clock, but recently we've been experimenting with using NTP.

We have one node (A) where you author time-sensitive events, and another node (B) that polls for pages of those events, loops through them and takes some action when they should occur. Without both nodes agreeing somewhat on the time you'll end up missing events.

Re:One second per second is harder than it sounds by superpatosainz on Tuesday July 22, 2014 @08:42AM (#8068598)
IRC: everything is timestamped to the subsecond. Any drifting causes channel takeovers and "random" nick kills, even links fail. Also, since a message from a server is broadcasted across the entire network, one misconfigured server can cause chaos in your hubs or completely unrelated leafs.

Or at least that's how I remember my experience managing an unstable (as in ddosed, every irc service in active development, a few in-house bots, etc) IRC network and having read the IRC and IRCv3 specs and unreal's and TS6 server message specs. If I'm wrong I'm sure someone here could correct me.

Re:One second per second is harder than it sounds by fanf2 on Wednesday July 23, 2014 @02:44AM (#8073182)
Your comment is very surprising. I don't see how clock drift can cause the problems you describe. There is no timestamping in the IRC protocol.

You can ask for a server's local time, but that is just for information.

There is a recently-added timestamp extension, but that seems to be to correct for delays so clients can record logs correctly. It is not for keeping the IRC network together.

Re:One second per second is harder than it sounds by superpatosainz on Wednesday July 23, 2014 @11:05AM (#8074720)
Yes, there is. Just not in the client side, it's server-to-server, and due to the centralised nature of IRC, it's critical.

For example, here's a paragraph from the UnrealIRCd server protocol (http://www.unrealircd.com/files/docs/technical/serverprotoco...):

> Unreal is very time-dependant. Users and channels, for example, are timestamped, and if server clocks are not synchronized properly, things can go very wrong very fast. See http://vulnscan.org/UnrealIrcd/faq/#67 for more information on this. Note that there is a slight difference between server time and what is actually reported by the UNIX date command or by the C time() function.

The other major side of IRC server protocol is the TS6 protocol (You can see it by yourself at http://irc-wiki.org/IRCd_Comparison). TS stands for TimeStamp, and you can see how critical timestamps are at https://github.com/avenj/irc-server-pluggable/blob/master/do... and https://github.com/atheme/charybdis/blob/master/doc/technica.... You can see TS are what control almost everything related to nicks, channels and their properties.

Before the implementation of TimeStamps and Services, it was really easy to abuse netsplits to take over channels and people's nicks.

There's a complete explanation at http://codeidol.com/internet/irc/The-IRC-Protocol/Timestamp-... (and of course, a Wikipedia link: http://en.wikipedia.org/wiki/Internet_Relay_Chat#Timestampin...).

Re:One second per second is harder than it sounds by dsymonds on Tuesday July 22, 2014 @12:05AM (#8067342)
Several security protocols (e.g. Kerberos) depend on systems having a somewhat reliable clock. They'll issue some sort of authentication token on system A that a user is meant to present to system B; the token has some expiry time, and the security of the system depends on systems A and B having comparable clocks.
Re:One second per second is harder than it sounds by tmarthal on Tuesday July 22, 2014 @12:08AM (#8067349)
I know that offhand gaming and financial services rely on very accurate clocks at the application level. And you can't necessarily rely on the clients to tell you what their time is, as it can be spoofed.
Re:One second per second is harder than it sounds by Arelius on Tuesday July 22, 2014 @04:06AM (#8067825)
Gaming is actually a really bad example since as you say, you can't really on the clients telling you a time as it may be spoofed. Thus the clients times are always ignored, and thus time sync is irrelevant.

I'll say that in about 10 years of working on multi-player games, I've never once put a wall, or other absolute time on the wire for any purpose than just printing server time to the player for informational purposes.

Re:One second per second is harder than it sounds by jacquesm on Tuesday July 22, 2014 @10:21AM (#8069104)
Time sync is very important because such games are typically played across clusters of machines, not just a single machine.
Re:One second per second is harder than it sounds by Arelius on Wednesday July 23, 2014 @02:42AM (#8073172)
If by such games you mean very specifically MMO's, then sure, I'll concede that time sync may be more valuable, but even then event serialization is normally done via an authority hand-off model that does not actually require time sync.

However, if you mean that all multiplayer games are typically played across "clusters of machines" and thus require time-sync then my original point addressed that.

Re:One second per second is harder than it sounds by rayiner on Tuesday July 22, 2014 @12:07PM (#8069682)
Wireless networking protocols may very precise time synchronization at the PHY/MAC layer. Generally, wireless technologies like WiMAX or LTE divide time into frames (e.g. LTE is 10 ms). Then depending on whether you're frequency duplexing (FDD) or TDD (time duplexing), the frames are divided into subframes with defined meaning. If different radios on the network don't agree as to frame start/end times, interference occurs.

Here's an interesting paper describing the software implementation of certain features that are usually handled by the hardware (well, by software running on the baseband processor): http://people.freebsd.org/~sam/FreeBSD_TDMA-20090921.pdf.

Re:One second per second is harder than it sounds by jrockway on Tuesday July 22, 2014 @01:58AM (#8067556)
I've used clockspeed. tlsdate + clockspeed turns out to be pretty reliable, and a lot less code than ntpd.
Re:One second per second is harder than it sounds by jfoutz on Monday July 21, 2014 @11:57PM (#8067319)