There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level.
This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casually streaming SPC data or Netflix over the same wire.
This is also in conjunction with the newer single pair Ethernet tech that the Automotive world is also interested in.
For anyone wondering:
> In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1.
> Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19]
* https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Sin...
> The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD).
* https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL
100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space.
* https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-au...
There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.
The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.
Doing a quick search, the documentation for Cisco, Arista, and Juniper all mention unicast PTP, so it may be the feature is becoming more prevalent.
See also "Enterprise Profile for the Precision Time Protocol With Mixed Multicast and Unicast Messages":
* https://datatracker.ietf.org/doc/html/draft-ietf-tictoc-ptp-...
Also, there are different types of PTP transparent clocks. They can either be end-to-end or peer-to-peer, and either one-step or two-step clocks. To be useful for NTP, I think it would need to be an end-to-end transparent clock and ideally it would be a one-step clock to avoid dealing with with the follow-up messages.
https://github.com/facebook/fboss/blob/master/fboss/agent/hw...
They have O(thousands) of switches (source: https://engineering.fb.com/2016/10/18/data-center-engineerin...) so IIRC they get the per-switch cost down by doing a custom design to reduce BOM. I believe the specs and designs are available under the Open Compute project here (search for "Facebook"): https://www.opencompute.org/wiki/Networking/SpecsAndDesigns
I could have used an in-between time unit, like hundredths of a millisecond, but that would have created embarrassingly long variable names.
What's the use case for microsecond-precise timekeeping, that is of interest to (i.e., benefits) FB?
But, by no mean, does it mean we use special timing - but now that I say that, I realize I have no clue where our time signal comes from, clearly it wouldn't be the internet because our trading servers aren't exactly directly connected there, we must have an internal ntp sever o_O
Imagine super high speed, high precision robotics, maybe a synthetic fiber layup machine or an exotic milling machine, that moves at 10m/s. If you have 1us of precision and accuracy then you can send movement commands that are precise down to 10um.
An example that does - I work for IceCube, a big neutrino detector at the South Pole. The detector is essentially a time of flight system that detects emitted light as it propagates through the ice when a neutrino interaction happens. We need good time distribution in the system so we can marry up hits from the 5000 odd sensors in the ice (and successive hits may be nanoseconds apart), and we need good absolute precision because we collaborate with other observatories.
Also pretty much all radio telescopes that do long baseline interferometry need excellent absolute timing. The South Pole Telescope has (I think) the most accurate clock on station, a hydrogen maser-based system. We use GPS.
My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive.
And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch.
I wonder how Facebook are connecting their precise new time servers to their regular servers?