Gah, just when you think you can trust time.nist.gov

Suggestions from the community for more reliable alternatives?

> Gah, just when you think you can trust time.nist.gov

You still can...

If you're that considered about 5 microseconds: Build your own Stratum 1 time server https://github.com/geerlingguy/time-pi

or just use ntppool https://www.ntppool.org/en/

It sounds like GPS, and thus a GPS-based stratum 1 server, uses these time servers, but they were successfully failed over:

> Jeff finished off the email mentioning the US GPS system failed over successfully to the WWV-Ft. Collins campus. So again, for almost everyone, there was zero issue, and the redundancy designed into the system worked like it's supposed to.

So failures in these systems are potentially correlated.

The author mentions another solution. Apparently he runs his own atomic clock. I didn’t know this was a thing an individual could do.

> But even with multiple time sources, some places need more. I have two Rubidium atomic clocks in my studio, including the one inside a fancy GPS Disciplined Oscillator (GPSDO). That's good for holdover. Even if someone were jamming my signal, or my GPS antenna broke, I could keep my time accurate to nanoseconds for a while, and milliseconds for months. That'd be good enough for me.

The CSACs that I have in a couple devices are 'atomic', and use Rubidium, but they're a bit lower accuracy than Cesium clocks [1] or Hydrogen Masers [2].

There are a few folks on the time-nuts mailing list who own such exotic pieces of hardware, but those are pretty far out of reach for most!

[1] https://www.microchip.com/en-us/products/clock-and-timing/co...

[2] https://www.microchip.com/en-us/products/clock-and-timing/co...

Atomic clocks cover a pretty big range of performance nowadays. You can pick up a used but serviceable rubidium frequency reference for a few hundred dollars but the difference between it and the top of the line clocks is almost as big as the difference between a it and a good pendulum clock.

Be aware that there are members of the NTP pool with less-than-honorable intentions and you don't get to pick-and-choose. Yes, they all should provide the time, but they also get your IP address.

For example: unlike the IPv4 space, the IPv6 space is too big too scan, so a number of "researchers" (if you want to call them that) put v6-capable NTP servers in the NTP pool to gather information about active v6 blocks to scan/target.

Do you have any acticles or references about this? That would be great research (pun intended) to find out

Is this one of those extraordinary claims that requires evidence? Or is it generally true that there are homey-pots in many of these services (NTP, mirrors, etc)

I'm more concerned about what you think they did to earn your trust in the first place

their handling it responsibly seems like more evidence for trusting them, not less?

Most places that need accurate time get it from GPS. That is 10-100 ns.

Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers.

Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.

Use the other servers as well: https://tf.nist.gov/tf-cgi/servers.cgi

For instance, time-a-wwv.nist.gov.

One should configure a number of different NTP sources instead of just a single host.

Yes.

Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS.

(If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)

More discussion:

NTP at NIST Boulder Has Lost Power

https://news.ycombinator.com/item?id=46334299

From NPR (22 points) https://news.ycombinator.com/item?id=46351105

I'm missing the nuance or perhaps the difference between the first scenario where sending inaccurate time was worse than sending no time, versus the present where they are sending inaccurate time. Sorry if it's obvious.

The 5us inaccuracy is basically irrelevant to NTP users, from the second update to the Internet Time Service mailing list[1]:

To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.

[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...

> Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet

How do those other applications obtain the precise value they need without encountering the Internet issue?

A lot of organizations also colocate timing equipment near the actual clocks, and then have 'dark fiber' between their equipment and the main clock signals.

Then they disperse and use the time as needed.

According to jrronimo, they even had one place splice fiber direct between machines because couplers were causing problems! [1]

[1] https://news.ycombinator.com/item?id=46336755

If I put my machine near the main clock signal, I have one clock signal to read from. The comment above was asking about how to average across many different clocks, presumably all in different places in the globe? Unless there's one physical location with all of the ones you're averaging, you're close to one and far from all the others so how is it done without the internet?

> How do those other applications obtain the precise value they need without encountering the Internet issue?

They do not use the Internet: they use local (GPS) clocks with internal high-precision clocks for carry-over in case GNSS signal is unavailable:

* https://www.ntp.org/support/vendorlinks/

* https://www.meinbergglobal.com/english/products/ntp-time-ser...

* https://syncworks.com/shop/syncserver-s650-rubidium-090-1520...

* https://telnetnetworks.ca/solutions/precision-time/

TIL/remembered GNSS satellites have onboard atomic clocks. Makes a lot of sense, but still pretty cool. Something like this, I guess?

https://en.wikipedia.org/wiki/Rubidium_standard

Yes, either Rb, Cs, or H standards depending on which GNSS system you're using.

For the most critical applications, you can license a system like Fugro AtomiChron that provides enhanced GNSS timing down to the level of a few nanoseconds. There are a couple of products that do similar things, all based on providing better ephemerides than your receiver can obtain from the satellites themselves.

You can get AtomiChron as an optional subscription with the SparkPNT GPSDO, for instance (https://www.sparkfun.com/sparkpnt-gnss-disciplined-oscillato...).

> You can get AtomiChron as an optional subscription with the SparkPNT GPSDO, for instance (https://www.sparkfun.com/sparkpnt-gnss-disciplined-oscillato...).

That's one hell of a healthy profit margin there O.o

The SiTime MEMS oscillator is about 100€ for one single chip, the mosaic-T GPS receiver is about 400€. Add 50€ for the rest (particularly the power input section looks complicated) and 50€ for handling, probably 600€ in hardware cost... sold for 2.500€.

The real money I think went into certification and R&D for a low-volume product - even though most of the hard work is done by the two ICs, getting everything orchestrated (including the PCB itself) to perform to that level of accuracy is one hell of a workload.

Yep, for hardware sold in low volumes, what looks like a healthy or even abusive profit margin turns out to be barely adequate to justify getting out of bed in the morning.

There are so many artificial bureaucratic barriers to entry these days, and it's not getting better.

If those other applications use their own local GPS clocks, what is the significance of NIST (and the 5μs inaccuracy) in their scenario?

GPS gets its time from NIST (though during this incident they failed over to another NIST site, so it wasn't impacted).

That is not correct at all. How did you arrive at that conclusion?

GPS has its own independent timescale called GPS Time. GPS Time is generated and maintained by Atomic clocks onboard the GPS satellites (cesium and rubidium).

I think GP might’ve been referring to the part of Jeff’s post that references GPS, which I think may be a slight misunderstanding of the NIST email (saying “people using NIST + GPS for time transfer failed over to other sites” rather than “GPS failed over to another site”).

The GPS satellite clocks are steered to the US Naval Observatory’s UTC as opposed to NIST’s, and GPS fails over to the USNO’s Alternate Master Clock [0] in Colorado.

[0] https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-N...

I find this stuff really interesting, so if anyone's curious, here's a few more tidbits:

GPS system time is currently 18s ahead of UTC since it doesn't take UTC's leap seconds into account [0]

This (old) paper from USNO [1] goes into more detail about how GPS time is related to USNO's realization of UTC, as well as talking a bit about how TAI is determined (in hindsight! - by collecting data from clocks around the world and then processing it).

[0] https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-N... [1] https://ntrs.nasa.gov/api/citations/19960042620/downloads/19...

It has its own timescale, but that still traces back to NIST.

In particular, the atomic clocks on board the GPS satellites are not sufficient to maintain a time standard because of relativistic variations and Doppler effects, both of which can be corrected, but only if the exact orbit is known to within exceeding tight tolerances. Those orbital elements are created by reference to NIST. Essentially, the satellite motions are computed using inverse GPS and then we use normal GPS based on those values.

> It has its own timescale, but that still traces back to NIST.

GPS gets its time from the US Naval Observatory:

> Former USNO director Gernot M. R. Winkler initiated the "Master clock" service that the USNO still operates,[29][30] and which provides precise time to the GPS satellite constellation run by the United States Space Force. The alternate Master Clock time service continues to operate at Schriever Space Force Base in Colorado.

* https://en.wikipedia.org/wiki/United_States_Naval_Observator...

The USNO does not seem to sync with NIST:

> As a matter of policy, the U.S. Naval Observatory timescale, UTC(USNO), is kept within a close but unspecified tolerance of the international atomic timescale published by the Bureau International des Poids et Mesures (International Bureau of Weights and Measures [BIPM]) in Sevres, France. The world's timing centers, including USNO, submit their clock measurements to BIPM, which then uses them to compute a free-running (unsteered) mean timescale (Echelle Atomique Libre [EAL]). BIPM then applies frequency corrections ("steers") to EAL, based on measurements from primary frequency standards and intended to keep the International System's basic unit of time, the second, constant. The result of these corrections is another timescale, TAI (Temps Atomique International or International Atomic Time). The addition of leap seconds to TAI produces UTC. The world's timing centers have agreed to keep their real-time timescales closely synchronized ("coordinated") with UTC. Hence, all these atomic timescales are called Coordinated Universal Time (UTC), of which USNO's version is UTC(USNO).

* https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-N...

The two organizations do seem to keep an eye on each other:

> The United States Naval Observatory (USNO) and the National Institute of Standards and Technology (NIST) make regular comparisons of their respective time scales. These comparisons are made using GPS common-view measurements from up to approximately 10 GPS satellites. The table below lists recent differences between the two time scales.

* https://www.nist.gov/pml/time-and-frequency-division/time-se...

> If those other applications use their own local GPS clocks, what is the significance of NIST (and the 5μs inaccuracy) in their scenario?

Verification and traceability is one reason: it's all very well to claim you're with-in ±x seconds, but your logs may have to say how close you are to the 'legal reality' that is the official time of NIST.

NIST may also send out time via 'private fibre' for certain purposes:

* https://en.wikipedia.org/wiki/White_Rabbit_Project

'Fibre timing' is also important in case of GNSS signal disruption:

* https://www.gpsworld.com/china-finishing-high-precision-grou...

If you must use the internet, PTP gets closer.

Alternate sources include the GPS signal, and the WWVB radio signal, which has a 60kHz carrier wave accurate to less than 1 part in 10^12.

Can you do PTP over the internet? I have only seen it in internal environments. GPS is probably the best solution for external users to get time signals with sub-µs uncertainties.

It's a good question, and I wondered the same. I don't know, but I'd postulate:

As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything.

When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get.

If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc.

Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it.

In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...

> […] where sending inaccurate time was worse than sending no time […]

When you ask a question, it is sometimes better to not get an answer—and know you have not-gotten an answer—then to get the wrong answer. If you know that a 'bad' situation has arisen, you can start contingency measures to deal with it.

If you have a fire alarm: would you rather have it fail in such a way that it gives no answer, or fail in a way where it says "things are okay" even if it doesn't know?

Has anyone here ever needed microsecond precision? Would love to hear about it.

I worked at Altera (FPGA supplier) as the Ethernet IP apps engineer for Europe for a few years. All the big telecoms (Nokia, Ericsson, Cisco, etc) use Precision Time Protocol (PTP) in some capacity and all required clocks to be ns levels for accuracy. Sometimes as low a 10ns at the boundary. Any imperfection in the local clock directly converts into timestamp error, and timestamp error is what limits PTP synchronization performance. Timestamps are the fundamental observable in PTP. Quantization and jitter create irreducible timestamp noise. That noise directly limits offset and delay estimation. Errors accumulate across network elements and internal clock error must be much smaller than the system requirement.

I think most people would look at the error and think "what's the big deal" but at all the telecoms customers would be scrambling to find a clock that hasn't fallen out of sync.

We don't use NTP, but for robotics, stereo camera synchronization we often want the two frames to be within ~10us of eachother. For sensor fusion we then also need a lidar on PTP time to be translated to the same clock domain as cameras, for which we also need <~10us.

We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.

> We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.

This doesn't seem right to me. NTP with default settings should be monotonic. So no jumps. If you disable it Linux enters 11-minute mode, IIRC, and that may not be monotonic.

Pedantically, a monotonic function need not have a constant first derivative. To take it further, in mathematics it is accepted for a monatomic function to have a countable number of discontinuities, but of course in the context of a digital clock that only increments in discrete steps, that’s of little bearing.

But that’s all besides the point since most sane time sync clients (regardless of protocol) generally handle small deviations (i.e. normal cases) by speeding up or slowing down the system clock, not jumping it (forward or backward).

You are correct, NTP prefers to jump first (if needed) and then slew afterwards (which is exactly what we want!), although it can jump again if the offset is too large.

In our case the jumps where because we also have PTP disciplining the same system clock, when you have both PTP and NTP fighting over the same clock, you will see jumping with the default settings.

For us it was easier to just do a one time NTP sync at the beginning/boot, and then sync the robots local network with only PTP afterwards.

In your stereo camera example, are these like USB webcams or something like MIPI CSI attached devices?

MIPI CSI, with USB cameras I usually am getting jitter in the millisecond range.

For a low precision environment to avoid sudden jumps I used SetSystemTimeAdjustment on Windows (now SetSystemTimeAdjustmentPrecise) to smoothly steer the system clock to match the GPS supplied time signal.

On Linux I think the adjtimex() system call does the equivalent https://manpages.ubuntu.com/manpages/trusty/man2/adjtimex.2....

It smears out time differences which is great for some situations and less ideal for others.

Also I implemented a simple PID controller that could automatically compensate against some types of internal clock drift.

Worked really well for the project.

Avoiding time jumps was really worthwhile.

We use nanosecond precision driven by GPS clocks. That timestamp in conjunction with star tracker systems gives us reliable positioning information for orbital entities.

https://en.wikipedia.org/wiki/Star_tracker

(Assuming "precision" really meant "accuracy") The network equipment I work on requires sub microsecond time sync on the network for 5G providers and financial trading customers. Ideally they'd just get it from GPS direct, but that can be difficult to do for a rack full of servers. Most of the other PTP use cases I work with seem to be fine with multiples of microseconds, e.g. Audio/Video over the network or factory floor things like PLCs tend to be find with a few us over the network.

Perhaps a bit more boring than one might assume :).

At a previous role, we needed nanosecond precision for a simulcast radio communications system. This was to allow for wider transmission for public safety radio systems without having to configure trunking. We could even adjust the delay in nanoseconds to move the deadzones away from inhabited areas.

We solved this by having GPS clocks at each tower as well as having the app servers NTP with each other. The latter burned me once due to some very dumb ARP stuff, but that's a story for another day.

We need nanosecond precision for trading - basically timestamping exchange/own/other events and to measure latency.

Yes, but always got it from GPS so presumably they'd be off about the same amount.

Distributed sonar, allows placing receivers willy-nilly and aligning the samples later.

Remote microphone switching - though for this you wouldn't notice 5us jitter, it's just that the system we designed happened to have granularity that good.

You probably want to ask about accuracy. Any random microcontroller from the 90s needs microsecond precision.

High speed finance is msec and below. Fastest publically known tick to trade is just shy of 14 nanos.

Timekeeping starts to become really hard, often requiring specialized hardware and protocols.

The high frequency trading guys

edit: also the linked slides in TFA

I believe LTE and 5G networks require it to coordinate timeslots between overlapping cells. Of course, they can use whatever reference they want, as long as all the cells are using the same one - it doesn't have to be UTC. Some (parts of) networks transmit it across the network, while others have independent GPS receivers at each cell site.

Synchronization is also required for SDH networks. Don't know if those are still used.

Someone else referenced low power ham radio modes like WSPR, which I also don't know much about, but I can imagine they have timeslots linked to UTC and require accuracy. Those modes have extremely low data rates and narrow bandwidths, requiring accurate synchronization. I don't know if they're designed to self-synchronize, or need an external reference.

When multiple transmitters are transmitting the same radio signal (e.g. TV) they might need to be synchronized to a certain phase relationship. Again, don't know much about it

WSPR doesn't require tight synchronization, but it does require pretty stable frequency sources over periods of 10s of seconds.

It is very common to integrate a GPS in a WSPR beacon to discipline the transmit frequency, but with modest thermal management, very ordinary crystal oscillators have very nice stability.

A database like Google Spanner has higher latency in proportion to the uncertainty about the time. Driving the time uncertainty down into the microsecond range, or lower, keeps latency low.

Telecom.

Precision Time Protocol gets you sub-microsecond.

https://en.wikipedia.org/wiki/Precision_Time_Protocol

Lots of things do. Shoot, even plain old TDM needs timing precision on the order of picoseconds to nanoseconds.

But does that require accurate UTC?

If you have a widely distributed system and you need tight synchronization, it can be much easier to regulate to an outside stable source than to try to make a stable consensus algorithm.

Yes, in some implementations.

I mean, we routinely benchmark things that take microseconds or less. I've seen a 300 picosecond microbenchmark (single cycle at 3GHz). No requirement that absolute time is correct, though.

Lightning detection. You have a couple of ground stations with known positions that wait for certain electromagnetic puses, and which record the timestamps of such pulses. With enough stations you can triangulate the location of the source of each pulse. Also a great way to detect nuclear detonations.

There is a german club that builds and distrubutes such stations (using GPS for location and timing), with a quite impressive global coverage by now:

https://www.blitzortung.org

Nuclear measurements, where the speed of a gamma ray flying across a room vs a neutron is relevant. But that requires at least nanosecond time resolution, and you’re a long way from thinking about NTP.

If you want sample accurate audio transmission you're going to want resolution on the order of 10s of microseconds.

How do you even get usable microsecond precision sync info from a server thousands of kilometers away? The latency is variable so the information you get can't be verified / will be stale the moment it arrives. I'm quite ignorant on the topic.

When we collected, correlated, and measured all controlling messages in a whole 4G network. Millisecond precision meant guaranteed out of order message flows.

Your speakers do so that people's voices match their mouth movements. The speaker clocks need to be in-sync with the cpu clocks and they operate at different frequencies.

Now I'm curious... How the hell do you synchronize clocks to such an extreme accuracy? Anybody have a good resource before I try to find one myself?

Look up PTP White Rabbit.

Thank you!

Are there any plans being made to prevent this happening in the future?

Yes, the US government is banning all those democrat windmills that conspired to blow over the NTP server.

??? The power outage was voluntary and surrounding towns chose not to turn off power. They could absolutely make infrastructure changes, and I’m sure the backups for power could make changes too.

It would greatly decrease electricity prices too. God knows how expensive it is to power those gigantic spinning things

I laughed audibly at that!

Out of curiosity, can anyone say the most impactful things they've needed incredibly accurate time for?

Spanner

(See https://docs.cloud.google.com/spanner/docs/true-time-externa...)

Does it need to be this close to NIST, or just relative to each other? Because the latter one is solved by PTP.

As far as I remember, near each other, but the comment I was replying to was specifically about needing accuracy.

It's been over a decade now since I managed the truetime team at google, things may have changed since :)

Spacecraft state vectors.

GPS

Not sure but synthetic massive aperture radio telescope would need syncing their local clocks.

I defer to the experts.

As far as I'm aware they just timestamp the sample streams based on a local gps backed reference. Then when they get the data/tapes in one computing center they can just run a more sophisticated correlation in software to smooth things out.

I work at a particle accelerator. We use White Rabbit (https://white-rabbit.web.cern.ch/) to synchronize some very sensitive devices, mostly the RF power systems and related data acquisition systems, down to nanosecond accuracy.

Telling people at the bar that I have an atomic clock at home.

cesium or rubidium?

rubidium + gps

As a very coarse number, 5µs is 1500 meters of radio travel.

If (and it isn't very conceivable) GPS satellites were to get 5µs out of whack, we would be back to Loran-C levels of accuracy for navigation.

Nitpick: UTC stands for Coordinated Universal Time. The ordering of the letters was chosen to not match the English or the French names so neither language got preference.

That doesn't quite match what the wikipedia page says:

> The official abbreviation for Coordinated Universal Time is UTC. This abbreviation comes as a result of the International Telecommunication Union and the International Astronomical Union wanting to use the same abbreviation in all languages. The compromise that emerged was UTC, which conforms to the pattern for the abbreviations of the variants of Universal Time (UT0, UT1, UT2, UT1R, etc.).

Follow the citation: https://www.nist.gov/pml/time-and-frequency-division/how-utc...

> ... in English the abbreviation for coordinated universal time would be CUT, while in French the abbreviation for "temps universel coordonné" would be TUC. To avoid appearing to favor any particular language, the abbreviation UTC was selected.

It also stands for Universel Temps Coordonné.

It's le temps universel coordonné in French.

That's an interesting rationale.

Reminds me of when a group is divided into two parts, dubbed group 1 and group A, such that neither feels secondary.

And if you have 3, group 1, group A, and group Alpha. Beyond that, just use colors.

The trick is making sure no one is happy with the final outcome

Universal Time, Coordinated.

This is how I say it in my head.

Now every time I see UTC, I hear the voice of Yoda: "Universal Time, Coordinated it is."

It's also the time in Iceland, conveniently

I found the most interesting part of the NIST outage post [1] is NIST's special Time Over Fiber (TOF) program [2] that "provides high-precision time transfer by other service arrangements; some direct fiber-optic links were affected and users will be contacted separately."

I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])? Telecom providers synchronizing 5G clocks for time-division duplexing [4]? Google/hyperscalers as input to Spanner or other global databases?

Seriously fascinating to me -- who would be a commercial consumer of NIST TOF?

[1] https://groups.google.com/a/list.nist.gov/g/internet-time-se...

[2] https://www.nist.gov/pml/time-and-frequency-division/time-se...

[3] https://www.finra.org/rules-guidance/rulebooks/finra-rules/4...

[4] https://www.ericsson.com/en/blog/2019/8/what-you-need-to-kno...

I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. Sub-microsecond latency, yes, but when that mattered it was in order to do something as soon as possible rather than as close as possible to Wall Clock Time X.

Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you.

The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.

> I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems.

mifid ii (uk/eu) minimum is 1us granularity

https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:...

It's 1 us granularity, which means you should report your timestamps with six figures after the decimal point.

The required accuracy (Tables 1 and 2 in that document) is 100 us or 1000 us depending on the system.

> The required accuracy (Tables 1 and 2 in that document)

no, Tables 1 and 2 say divergence, not accuracy

accuracy is a mix of both granularity and divergence

regardless, your statement before:

> The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.

is not true

> accuracy is a mix of both granularity and divergence

I respectfully disagree.

In context, "granularity" is nothing more than a resolution constraint on reported timestamps. Its inclusion adjacent to the specified "divergence from UTC" is a function of market manipulation surveillance objectives as discussed in preamble item (2), and really doesn't have anything to do with accuracy proper.

any time i am certain of something i never capitalize and i do not end my sentences with periods or use any punctuation because i like people to believe i am an omniscient narrator who cannot be interrupted

> mifid ii (uk/eu) minimum is 1us granularity

1us is nothing special for GPS/NTP/PTP appliances (especially with OCXO/rubidium oscillators):

* https://www.microchip.com/en-us/products/clock-and-timing/sy...

* https://www.meinbergglobal.com/english/productinfo/gps-time-...

My guess would be scientific experiments where they need to correlate or sequence data over large regions. Things like correlating gravitational waves with radio signals and gamma ray bursts.

those are GPS based too. You typically would have a circuit you trained off off 1PPS and hopefully had a 10 or so satellites in view.

You can get 50ns with this. Of course, you would verify at NIST.

GPS could be blocked easily, and AFAIK even given corrupted inputs. And HFT could possibly benefit from blocking or corrupting competitors GPS.

Deploying a GPS jammer in civilized territory is a great way to go to prison.

Would it actually go so far?

Would the police actually try to investigate from where came the jammer? Might the competing firm possibly even finance an investigation themselves privately? And if so, would the police then accept the evidence?

People have done far more evil things for money.

The victim firm would definitely notice, they’d tell the FCC, and their investigators will show up with a device that literally points them to wherever the jammer is. If you do this for stupid, silly reasons you will get fined[1], if you do it in commission of another crime you will probably get made an example of. It doesn’t matter how evil you are, it’s hilariously easy to get caught doing this.

[1]: https://www.nj.com/news/2013/08/man_fined_32000_for_blocking...

We are talking about the UK, not the US. And the jammer will most likely be tucked away in some closet with no hint as to how it got there.

Where were we talking about the UK? All anyone said in this message chain was HFT (and NIST).

Sorry, you are correct. As soon as the subject of HFT came up I was thinking about London and the things they do to reduce latency to the exchanges in North America. It's too late to edit or remove my previous message.

> “Mr. Bojczak claimed that he installed and operated the jamming device in his company-supplied vehicle to block the GPS … system that his employer installed in the vehicle,” the FCC decision stated.

I'm not surprised that somebody would try and do this. However it is just so stupid at every level.

Next to Newark Airport too. He’s lucky they didn’t throw the book at him - they could’ve hit him for reckless endangerment.

It's not like foreign adversaries care.

The parent was saying HFT firms would do this to other HFT firms. They would care about doing this kind of thing - it’s not a white collar crime. And foreign adversaries would care about doing this during peacetime, especially for very unclear benefit.

> ...and hopefully had a 10 or so satellites in view.

I believe you'll need 12 GPS sats in view to gain incremental accuracy improvement over 8.

> a commercial consumer

Where does it say these are commercial consumers?

https://en.wikipedia.org/wiki/Schriever_Space_Force_Base#Rol...

> Building 400 at Schriever SFB is the main control point for the Global Positioning System (GPS).

> I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])?

To start with, probably for scientific stuff, à la:

* https://en.wikipedia.org/wiki/White_Rabbit_Project

But fibre-based time is important in case of GNSS time signal loss:

* https://www.gpsworld.com/china-finishing-high-precision-grou...

I think Google uses chrony instead of NTP

Google doesn't use chrony specifically, just an algorithm that is somewhat chrony-like (but very different in other ways). It's called Google TrueTime.

Oh right. Their cloud-init script uninstalls NTP and installs chrony each time our VMs boot

Ah yeah. For VPS tenants it makes sense they would default to Chrony. They have a public facing pool of NTP servers at `time.google.com`, and for tenant use they provide `metadata.google.internal`, which is probably where the Chrony config file points. IIRC TrueTime is not actually open source and is only used internally on their infrastructure.

I'm sure all of that is true, but so is "Department of Defense".

They're also the largest holder of IPv4 space, still. https://bgp.he.net/report/peers#_ipv4addresses

Why does the DoD hold so many IPv4s?

They were assigned a huge prefix at the creation of the internet iirc

ARPANET, precursor to the internet, was a DoD project.

SIGINT as a source clock for others in a network doing super accurate TDOA for example.

But they do not need absolute time, and internal rubidium clocks can keep the required accuracy for a few days. After that, sync can be transferred with a portable plug, which is completely viable in tactical/operational level EW systems.

science equipment, distributed radio-telescopes where you need to precisely align data received at different locations

> Google/hyperscalers as input to Spanner or other global databases?

Think Google might have rolled their own clock sources and corrections.

Ex: Sundial, https://www.usenix.org/conference/osdi20/presentation/li-yul... / https://storage.googleapis.com/gweb-research2023-media/pubto... (pdf)

Only Boulder servers lost sync.

To say NIST was off is clickbait hyperbole.

This page: https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine.

However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody.

IMHO, most applications should use pool.ntp.org

Who does use those top-level servers? Aren’t some of them propagating the error or are all secondary level servers configured to use dispersed top-level servers? And how do they decide who is right when they don’t match?

Is pool.ntp.org dispersed across possible interference and error correlation?

You can look at who the "Stratum 2" servers are, in the NTP.org pool and otherwise. Those are servers who sync from Stratum 1, like NIST.

Anyone can join the NTP.org pool so it's hard to make blanket statements about it. I believe there's some monitoring of servers in the pool but I don't know the details.

For example, Ubuntu systems point to their Stratum 2 timeservers by default, and I'd have to imagine that NIST is probably one of their upstreams.

An NTP server usually has multiple upstream sources and can steer its clock to minimize the error across multiple servers, as well as detecting misbehaving servers and reject them ("Falseticker"). Different NTP server implementations might do this a bit differently.

I've been running servers for the pool for years. They are checked regularly for accuracy/uptime or it's score goes down in the pool and eventually gets removed. I sync from 5 stratum 1 servers and use chrony these days.

Facebook had a really interesting engineering blog about building their own timeservers: https://engineering.fb.com/2020/03/18/production-engineering...

Really well written for anyone who is interested.

From my own experience managing large numbers of routers, and troubleshooting issues, I will never use pool.ntp.org again. I’ve seen unresponsive servers as well as incorrect time by hours or days. It’s pure luck to get a good result.

Instead I’ll stick to a major operator like Google/Microsoft/Apple, which have NTP systems designed to handle the scale of all the devices they sell, and are well maintained.

I believe if you use time.nist.gov it round robins dns requests, so there’s a chance you’d have connected to the Boulder server. So for some people they would have experienced NIST 5 μs off.

Not exactly the topic of discussion but also not not on topic: just wanted to sing praise for chrony which has performed better than the traditional os-native NTP clients in our testing on a myriad of real and virtualized hardware.

Chrony is the default already in some distros (RHEL and SLES that I know of), probably for this very reason.

Maybe I missed something, but I don't quite understand the video title "NIST's NTP clock was microseconds from disaster". Is there some limit of drift before it's unrecoverable? Can't they just pull the correct time from the other campus if it gets too far off?

I'll consider it clickbait...

... unless someone with real experience needing those tolerances chimes in and explains why it's true.

I'd have thought Jeff to be above clickbait, but here we are

The web article title doesn't mention any disaster. I don't see the embedded youtube video - probably have to many privacy options on.

If Jeff wants to maintain his job of making Youtube videos, youtube itself forces him to make clickbait thumbnails and titles.

They have A/B testing infrastructure available to creators for exactly that purpose.

If you say "Oh but he doesn't have to use that" you are wrong. If a creator's videos get low enough click through and "engagement", youtube just stops showing it to people.

Jeff, like every creator on youtube, is in an antagonistic deathmatch for your attention with people like Mr. Beast, and Youtube would rather five Mr Beasts than a thousand Jeffs. Youtube builds their platform to empower and enrich Mr Beast, and to force everyone else to adopt more of Mr Beasts methods to stay getting views, and getting their paycheck

It does not matter if you are "subscribed" to a channel for example. Youtube will still fail to show you new videos from your subscriptions if you don't play the stupid game enough.

Stop blaming the people who did not at all make this choice. Blame Youtube. Support platforms, like nebula and whatever that guntube service is called, with real money.

Did you know this video existed before it was posted to HN? I did, because it showed up in my feed even though I am not subscribed to jeff, because he plays the game well. I watched it too because I had no idea that weather was causing such a problem in Colorado and love hearing about NIST and the people who basically run the infrastructure of the internet that everyone takes for granted.

I'm afraid you're preaching to someone who doesn't watch talking heads :)

My only interaction with Jeff's articles are through transcripts if they make it to the HN front page.

Too late to edit so...

>I watched it too because I had no idea that weather was causing such a problem in Colorado and love hearing about NIST and the people who basically run the infrastructure of the internet that everyone takes for granted.

Are you aware that text based news about the NIST problems have been posted on HN in the past few days?

You'll never guess why. The answer might shock you.

I know some HFT people who made a few hundred K off of this.

Can you elaborate on how? Did they consciously exploit this or their systems just had a lucky glitch?

I find this topic and thread fascinating.

I took too much Adderall today.