News for hackers. Stuff that matters.
Re:GPS error caused '12 hours of problems' for companies by madaxe_again on Tuesday February 09, 2016 @03:33PM (#11068528)
I'm curious as to why they're using atomic clocks in space for a high resolution time source, rather than an atomic clock on earth... Latency?
Re:GPS error caused '12 hours of problems' for companies by elfchief on Tuesday February 09, 2016 @04:42PM (#11069006)
The problem with atomic clocks in general is that they have very good short term stability, but not-so-great long-term stability. So they are very good at measuring the amount of time that has passed between two events, but not as great for telling you exactly what time an event happened.

The other problem is that there's not really such thing as a single, true time. Precision timekeeping is one of those places where relativity matters -- differences in altitude and the geology of the earth under your feet at a particular point change how fast any particular clock 'ticks'.

So to accommodate these differences, if you want events to be synchronized across multiple locations over a long period, you both need a single agreed-upon "current time", and some way to get information about that time information to the sites that need it.

You could surely do this with directly connected wires that were very carefully measured, but that gets expensive very quickly. The GPS system already requires super-precise time just to function, so it makes for a great distribution mechanism of "what time is it" information, where the people running the GPS system are bothered with all the details of figuring out and standardizing a single time, and you 'just' have to listen.

The GPS folks also keep track of things like how fast the GPS standard clock is drifting from the UTC standard clock (since no two clocks will agree over the long term, period). And this is actually where the problem happened -- Two of the parameters broadcast (A0 and A1) were incorrect, and it's these parameters that are used to specify the current time difference between the GPS master clock and the UTC master clock (the GPS master clock is disciplined to be as close to UTC as possible, but is always off by a minute amount, and that's what A0/A1 represent).

Pretty much it boils down to, much as it is in distributed systems, "there is no now". Even with maximum care and the highest quality clocks, there's really not a definition of 'now' that's consistent across more than a single location. GPS is currently the closest thing we have to a globally-distributed definition of 'now'.

Re:GPS error caused '12 hours of problems' for companies by planteen on Tuesday February 09, 2016 @07:44PM (#11070026)
Yep and "simultaneous" is not simple either as Einstein figured out in 1905: "So we see that we cannot attach any absolute signification to the concept of simultaneity, but that two events which, viewed from a system of co-ordinates, are simultaneous, can no longer be looked upon as simultaneous events when envisaged from a system which is in motion relatively to that system."

https://www.fourmilab.ch/etexts/einstein/specrel/www/

Re:GPS error caused '12 hours of problems' for companies by cnvogel on Wednesday February 10, 2016 @02:43AM (#11071288)

    > atomic clocks in general is that they have very good
    > short term stability, but not-so-great long-term
    > stability
Well... The normal number to look at is the "Allan Variance". Over a timescale of days (100'000 seconds) a good rubidium will achieve about 10^-12, which is 10ns. A hydrogen maser will have 10^-13 or 10^-14 over even longer timescales. This is pretty great long-term stability in my book. A good, but undisciplined, quartz oscillator will have 10^-12 over timescales of hours.

http://ivs.nict.go.jp/mirror/publications/gm2002/takahei/img... http://www.ke5fx.com/rb.htm http://www.thinksrs.com/assets/instr/PRS10/PRS10diag2LG.gif

By definition, anything steered by GPS will have infinite precision for infinitely long time-scales, as GPS time and UTC time ultimately are steered such that they are within a certain tolerance of each other.

http://www.leapsecond.com/pages/tbolt-tc/

And UTC time is the pretty elaborate average of ensembles of atomic clocks, hence atomic clocks produce the international reference time there is.

You can, of course, improve the long-term stability of any clock by locking it to a GPS (or other means of time-signal distribution). And for a good Caseium that might only require you to look at the phase, and slightly nudge a potentiometer (magnetic correction field adjustment) on the front by a tiny amount every week. But atomic clocks in general are pretty darn impressive.

UPDATE: This presentaiton has all the interesting plots combined, it seems: http://www.hipster.net/ShadNygren_Atomic_Clocks_for_Amateur_... (pages 45 and following).

Re:GPS error caused '12 hours of problems' for companies by elfchief on Wednesday February 10, 2016 @07:45PM (#11077455)
Nice presentation! Thanks for the link!

The particular nit you pick here is a fair one, if you take my comment literally, rather than as the (over?)simplification of a complex concept I was trying to summarize.

Yes, in absolute terms, an atomic clock will have excellent stability over the long term in its own frame of reference. As long as you don't need your atomic clock to match any other clock in the universe, the long-term stability is excellent.

The particular topic being discussed here, though, is equipment that is designed to synchronize events across a large area. Even atomic clocks that are literally 100% accurate will diverge if installed in different locations. So if you want your atomic clock to actually be able to tell you what time it is (in a way that matches up closely with anyone else), you have to constantly be adjusting it to match a common standard (e.g. UTC)

These adjustments are a form of instability, since to correct for changes, you have to make your seconds either faster or slower than they actually are locally, on a constantly-changing basis, or you have to step your time and have less (or more) than (Y - X) time actually elapse between timestamps X and Y.

As soon as you move beyond a single frame of reference, you can't have both short and long term stability.

I suppose I could have used a different word than 'stability', but that seemed to get across the underlying effect best. I suppose another way of saying it (that is more correct) is that atomic clocks are great for telling you how much time has passed (in your local reference frame), but not so great for telling you what time it is. GPS is great for telling you what time it is, but not so great for telling you how much time has passed (in your local reference frame).

(The best clocks for timekeeping, of course, are atomic clocks disciplined by a common reference like GPS, which gives you the option of deciding your own long term vs. short term tradeoff however you'd like. But the original question was "why do we need GPS at all, and not just atomic clocks?".)

Re:GPS error caused '12 hours of problems' for companies by eru on Tuesday February 09, 2016 @07:05PM (#11069854)
Google is also working on a globally distributed `now': http://research.google.com/archive/spanner.html

Of course, they make use of GPS, but also add their own atomic clocks.

Re:GPS error caused '12 hours of problems' for companies by elfchief on Tuesday February 09, 2016 @09:13PM (#11070327)
Even Google's "now" is still a little fuzzy -- their system doesn't so much define "now" as it defines the uncertainty around "now", which you can take advantage of for figuring out ordering. This paragraph (from section 1 of their paper) summarizes it pretty well:

> The key enabler of these properties is a new TrueTime > API and its implementation. The API directly exposes > clock uncertainty, and the guarantees on Spanner’s timestamps > depend on the bounds that the implementation provides. > If the uncertainty is large, Spanner slows down to > wait out that uncertainty.

...so really, GPS and atomic clocks aren't for defining "now" they're for providing a keeping the window of uncertainty around "now" to a small value. The value mentioned in the paper for this is "generally less than 10ms", which is pretty good for a large-scale distributed system, but massive in terms of precision timekeeping. It is, after all, an uncertainty that's three orders of magnitude larger than the GPS glitch that started this discussion...

Re:GPS error caused '12 hours of problems' for companies by chiph on Tuesday February 09, 2016 @03:43PM (#11068594)
A ground-based transmitter could be destroyed in an attack, so having the clocks on orbit meant they were protected and would still be operational for a retaliatory ICBM launch.

Edit: So far as latency - the on-board clocks get resynced today every now and again against a ground-based reference time as they pass overhead of a USAF transmitter (primary + backup). If they used only moderately-accurate clocks as a cost-saving measure (or were at the limit of late 1970's technology) drift would be more severe and the tail-end of each orbit would be less correct than at the start of the orbit. So additional ground stations would be required in parts of the planet not always so friendly to the US, and they would have to coordinate their time sources, allowing for the light-speed delay travelling around the earth. Much more expensive/harder solution from an operational aspect.

Edit Edit: Don't forget that the GPS system was designed around military requirements. All the satellites (even today) have nuclear weapon detonation detectors on board and are a primary input to NORAD.

> The 2nd SOPS has three missions: global navigation, time transfer and nuclear detection.

http://www.schriever.af.mil/GPS/

Re:GPS error caused '12 hours of problems' for companies by madaxe_again on Tuesday February 09, 2016 @04:23PM (#11068888)
Totally get it in the military context, and why it's useful in the context of a missile lobbing holiday, but was rather more curious as to why it'd be used as a time source for synchronising packets in commercial applications as the article describes.

Based on what others have said and my own inkling I think it's about synchronisation at a fine timescale without having to trouble yourself with bothersome shit like relativity, and leaving that particular pain in the ass to the US military. That's what I meant by latency, but phone keyboard, walking, lazy!

Re:GPS error caused '12 hours of problems' for companies by chiph on Tuesday February 09, 2016 @09:20PM (#11070352)
What it is is outsourcing your time source to the USAF. Aka introducing an external dependency.

If these people are upset over 13 microseconds, wait until the DoD turn selective availability back on during a war or start adjusting satellite positions to avoid incoming ASAT missiles.

Interesting fact: GPS time doesn't take leap seconds, etc. into account. As of last summer, it's ahead of UTC by 17 seconds.

http://tycho.usno.navy.mil/leapsec.html

Re:GPS error caused '12 hours of problems' for companies by kylegordon on Wednesday February 10, 2016 @05:56PM (#11076865)
There is an executive order to keep SA at zero. They have other means to deny GPS availability now, and it doesn't involve futzing global availability.
Re:GPS error caused '12 hours of problems' for companies by jewel on Tuesday February 09, 2016 @03:42PM (#11068584)
I think it's a matter of price. GPS receivers are cheap and widely supported by timekeeping software. The cheapest atomic clock I can find is $1,500, but it's a hard thing to search for since the term is used to refer to radio-synchronized clocks.

If you did have an atomic clock in every datacenter, you'd still need to make sure that they didn't develop a problem that caused them to drift apart, and also you'd need to make sure that they were in sync with each other in the first place.

Re:GPS error caused '12 hours of problems' for companies by madaxe_again on Tuesday February 09, 2016 @04:27PM (#11068909)
$1500 is nothing if you're a telco. If cost were the reason they wouldn't blink at having five caesium fountains in each location - but as another observed it's likely about synchronicity - relativity is a problem if you're looking to have two clocks on opposite sides of the planet in perfect sync.
Re:GPS error caused '12 hours of problems' for companies by ploxiln on Tuesday February 09, 2016 @04:08PM (#11068786)
it would be a real problem if they drifted at all ... that's kinda the point of atomic clocks, no?
Re:GPS error caused '12 hours of problems' for companies by mentat on Tuesday February 09, 2016 @05:44PM (#11069431)
Within their frame of reference. Many many frames exist.
Re:GPS error caused '12 hours of problems' for companies by ploxiln on Tuesday February 09, 2016 @08:20PM (#11070156)
That's true, but if it's sitting in your datacenter which is stably attached to some point on the surface of the earth, that's a well defined frame of reference.
Re:GPS error caused '12 hours of problems' for companies by detaro on Tuesday February 09, 2016 @04:28PM (#11068918)
Of course atomic clocks drift as well, and as soon as you run multiple you have to keep that in mind. GPS (or rather a GPS-controlled quartz) is probably better quality than cheap atomic clocks.

GPS is a convenient way to get well-synced precise time nearly everywhere you can see the sky.

Re:GPS error caused '12 hours of problems' for companies by ploxiln on Tuesday February 09, 2016 @08:23PM (#11070172)
GPS satellites themselves use atomic clocks

(and to find your position you use their also-broadcast orbital positions and math for the effects of relativity etc)

Re:GPS error caused '12 hours of problems' for companies by KMag on Tuesday February 09, 2016 @11:45PM (#11070816)
I think you missed the word "cheap" in the GP.
Re:GPS error caused '12 hours of problems' for companies by detaro on Tuesday February 09, 2016 @08:29PM (#11070191)
Yes, I'm aware. ?
Re:GPS error caused '12 hours of problems' for companies by ploxiln on Tuesday February 09, 2016 @09:50PM (#11070451)
GPS satellites use atomic clocks which don't drift aka don't become 0.1% slower or faster than they should be. OK, technically they drift, but not enough to be noticeable by anyone. They're the standard to which other clocks are compared.

According to https://en.wikipedia.org/wiki/Atomic_clock#Evaluated_accurac... existing atomic clocks can drift by 1 second in 100 million or 150 million years. You won't notice that in your high-speed datacenter applications over the course of a few decades.

(unless perhaps if one is plain broken, and it's noticed by comparing to the others and taken out of service)

Re:GPS error caused '12 hours of problems' for companies by varjag on Tuesday February 09, 2016 @03:42PM (#11068591)
GPS reference clocks are nanosecond resolution. Synchronizing from Earth would be challenging due to varying transmission distances and relativistic effects.
Re:GPS error caused '12 hours of problems' for companies by fennecfoxen on Tuesday February 09, 2016 @03:36PM (#11068547)
At a guess, they need a very small number of clocks (for uniformity) and they use the existing clocks in space because that signal distribution was more effective than new terrestrial infrastructure.