News for hackers. Stuff that matters.
Re:Clockwork raises $21M to keep server clocks in sync by evol262 on Friday March 18, 2022 @09:46PM (#30730081)
> “Currently, nobody uses time except for maybe Spanner at Google, CockroachDB or someone doing database things,” Rosenblum said. “We believe that there’s a lot more places, especially as more and more time-critical things came up. We can do time sync, since we figured out how to do that pretty well. And so we asked: is this part of a trend where we’re going to start programming these systems differently? And [researchers] got kind of excited about that possibility of us being able to pull this off.”

Who gave this guy who's apparently never heard of kerberos, ceph, SAML, or any other technology the rights to interview? Sure, their windows are larger, but "nobody uses time" is so clueless it blows the mind, and it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea

Re:Clockwork raises $21M to keep server clocks in sync by paxys on Friday March 18, 2022 @10:08PM (#30730226)
None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem. So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense.

If these folks can achieve what they want at a couple orders of magnitude cheaper than current prices then you are absolutely going to see a lot more regular use cases show up to take advantage of it.

Re:Clockwork raises $21M to keep server clocks in sync by bitcharmer on Saturday March 19, 2022 @02:13AM (#30731577)
> None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem

No, not really. I work in finance (HFT, systematic market making) and this industry heavily relies on high precision clock sync. It's even regulated by law (MiFID II); in practice everyone on the street uses PTP because if you don't know how fast you're really going or where your latency spikes you already lost the race, just don't know about it.

There's a couple of other domains where micro- and nanosecond level time sync is of paramount importance. PTP has become much cheaper over the years especially when you are in a data centre and can get it as a service instead of setting up grand-masters, slaves, GPS antennas, etc.

This statement just reads like badly researched topic on their part.

Re:Clockwork raises $21M to keep server clocks in sync by lrhegeba on Saturday March 19, 2022 @01:45PM (#30735851)
especially the Huygens-implementation delivering accuracy in nanoseconds could be some part of its secret sauce (https://en.wikipedia.org/wiki/Clock_synchronization#Huygens)
Re:Clockwork raises $21M to keep server clocks in sync by pclmulqdq on Saturday March 19, 2022 @12:43AM (#30731112)
They can't do better than a microsecond or so without hardware timestsmps. There aren't a lot of applications that seem to need anything between milliseconds and nanoseconds. MiFID II did open a nice market in finance, but many of those companies use PTP anyway.

I considered starting a competitor in 2018 when I first saw this company, but I don't have the same connections to customers that these guys do.

My conclusion was that PTP precision with software timestamps would be a good company, but not something between. I hope they can prove me wrong!

Re:Clockwork raises $21M to keep server clocks in sync by dan-robertson on Saturday March 19, 2022 @07:20AM (#30732911)
My understanding is that PTP is neither necessary nor sufficient. The main advantage of it is that some hardware will only support adding hardware timestamps to PTP messages (ie you want to use the time packets left/arrived at your network card rather than the time you processed them in the kernel/userspace). But I think the specialness of PTP can lead to dealing with a lot of bugs from bad implementations.
Re:Clockwork raises $21M to keep server clocks in sync by Gelob on Friday March 18, 2022 @10:24PM (#30730319)
Telco workloads and 5g require precise time using technologies like ptp. The cheaper, more commodity those solutions get the better.
Re:Clockwork raises $21M to keep server clocks in sync by _3u10 on Saturday March 19, 2022 @12:26AM (#30731006)
If only these devices had a clock capable of measuring time to within 1 nanosecond.

If one has such an accurate clock they might be able to accurately plot their position on earth to within 1 ft merely by checking the timed pulses of orbiting satellites.

Re:Clockwork raises $21M to keep server clocks in sync by jazzyjackson on Saturday March 19, 2022 @05:17AM (#30732341)
I guess you're downvoted for being sarcastic or maybe people don't know 20 dollar GPS chips provide a ~10 nanosecond resolution clock
Re:Clockwork raises $21M to keep server clocks in sync by SAI_Peregrinus on Saturday March 19, 2022 @09:44AM (#30733641)
And GPS-disciplined oven controlled crystal oscillators (OCXOs) provide some of the best frequency accuracy and stability around. They range from about a hundred dollars up through a few thousand, depending on phase noise.
Re:Clockwork raises $21M to keep server clocks in sync by _3u10 on Saturday March 19, 2022 @02:04PM (#30736050)
Haha I have no idea. But yeah just call the GPS algo a Time AI and voila, $21 m in funding to go get some kids in shenzen to put a GPS chip on a USB, PCIe, NVME slot.

Hey look we got a GPU to learn the Kalman filter algo.

Many are actually better than 10 ns basically a 1 foot CEP is 1 ns. If they are fixed and always on the accuracy it can build over a day is incredible.

Re:Clockwork raises $21M to keep server clocks in sync by jazzyjackson on Sunday March 20, 2022 @01:37PM (#30744713)
granted it does need an unobstructed view of the sky, so getting it on a PCI-e board would be an innovation (:
Re:Clockwork raises $21M to keep server clocks in sync by krisoft on Saturday March 19, 2022 @10:07AM (#30733796)
> So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense.

If you arrive to the railway station 10 minutes +-2 minutes before your train leaves you are using time. In a very real sense. In robotics we sync multiple computers on the robot to about miliseconds, that is using time. In a real sense.

Maybe more accurate, cheaper sync will enable more applications. Maybe it is a good business to specialise on it. But saying that nobody excep for Google, CockroachDB or someone doing database things uses time is bulshit and can be called out for what it is.

Re:Clockwork raises $21M to keep server clocks in sync by ctvo on Saturday March 19, 2022 @10:59AM (#30734279)
> If you arrive to the railway station 10 minutes +-2 minutes before your train leaves you are using time. In a very real sense. In robotics we sync multiple computers on the robot to about miliseconds, that is using time. In a real sense.

Why are you in a very specific technical discussion correcting what is obvious? Yes, you're using time. You're not using time at the precision this company is targeting. At that precision, there are few current use cases, but if they make it cheaper, there will be more. What's there to argue here? You're offended you're not being included as users of time?

"We're lunching more accurate satellite imaged maps. The current users are largely nation states and niche industries due to its cost. This will enable more map use cases"

I use Google Maps all the time. I used it for my last trip! How dare you.

Re:Clockwork raises $21M to keep server clocks in sync by krisoft on Saturday March 19, 2022 @05:46PM (#30737845)
> You're offended you're not being included as users of time?

Please refrain from personal attacks. It is unecessary and doesn’t add to the conversation. Thank you.

> I use Google Maps all the time. I used it for my last trip! How dare you.

Do note that your example didn’t say “nobody uses maps”. If an imaginary salesperson would say “nobody uses maps” I would call bulshit on that too.

> Yes, you're using time.

Great, so the company representative should simply not say “nobody is using time”. And the commenter I was responding to shouldn’t say that applications which are fine with a lower accuracy “aren't ‘using time’ in any real sense”.

Heck, all I know maybe nanosecond accuracy is where the bees knee is, and everyone is going to be amazed by the awesome new applications it is going to enable. But you don’t need to disregard all the history of timekeeping and all the current applications to make that point.

On 28 April 1789 a mutiny broke out on HMS Bounty. The ship’s former captain, departing on a rowboat, demanded that the mutineers give him K2 the ship’s chronometer. The mutineers refused, because the chronometer was worth about as much as the whole ship, and they needed it for their onward navigation. Somebody should have told them that they are not using time in any real sense. Probably they would have laughed at that idea.

Re:Clockwork raises $21M to keep server clocks in sync by gnufx on Saturday March 19, 2022 @10:09AM (#30733811)
What are those applications? They've passed me by unless that's meant to cover large-scale physics measurements of some sort. I don't understand how it would even make sense in mainstream supercomputers. It should be good to synchronize OS scheduling in HPC nodes to minimize jitter in tightly-coupled applications, but that's not ns stuff and I haven't come across it actually being done (though Bull had some support as free software).

I don't see how something like Kerberos or TOTP, that fails without some sort of synchonization, can be defined as not "using time" in a real sense. (If cluster nodes were drifting by seconds, I'd be checking them, even if that was only going to affect something like make.)

Re:Clockwork raises $21M to keep server clocks in sync by ignoramous on Saturday March 19, 2022 @10:55AM (#30734238)
Google uses precision time (SunDial) for observability, telemetry, congestion control too: https://www.youtube.com/watch?v=GEXP68yNPPM
Re:Clockwork raises $21M to keep server clocks in sync by Saris on Friday March 18, 2022 @10:46PM (#30730434)
Makes me curious how accurate a GPS clock is, they must be pretty good given the timing precision needed for accurate positioning.
Re:Clockwork raises $21M to keep server clocks in sync by WinterMount223 on Friday March 18, 2022 @11:03PM (#30730545)
For GPS precision is much more important than accuracy.
Re:Clockwork raises $21M to keep server clocks in sync by Dylan16807 on Saturday March 19, 2022 @12:05AM (#30730886)
Precision vs. accuracy of what measurement?

For GPS to work at all, you need very accurate knowledge of the offset between each satellite's signal.

The only way I can think of to have precision but not accuracy is if you design a system where there's a buffer between the antenna and the signal processing and you don't know how long the buffer is. Is a design like that a practical concern?

Re:Clockwork raises $21M to keep server clocks in sync by karpierz on Saturday March 19, 2022 @12:08PM (#30734919)
They're saying that as a receiver of GPS signals, it doesn't matter if your clock is out of sync with everyone else's clocks, as long as it measures the length of a nanosecond accurately.
Re:Clockwork raises $21M to keep server clocks in sync by Dylan16807 on Saturday March 19, 2022 @02:09PM (#30736123)
But GPS gives you a clock signal.
Re:Clockwork raises $21M to keep server clocks in sync by WinterMount223 on Saturday March 19, 2022 @10:58AM (#30734271)
Of UTC. Theoretically GPS can work out of sync with respect to UTC as long as all satellites are in sync among themselves.
Re:Clockwork raises $21M to keep server clocks in sync by Dylan16807 on Saturday March 19, 2022 @02:14PM (#30736170)
Oh, you're talking about a theoretical situation where the GPS satellites are reprogrammed to use a different synchronization source? Personally I'd still call that highly accurate, but to a different time standard. But I see where you're coming from.
Re:Clockwork raises $21M to keep server clocks in sync by ahazred8ta on Saturday March 19, 2022 @12:42AM (#30731108)
With specialized GPS receivers, about +/-2ns. Closer to 40ns with commercial receivers.
Re:Clockwork raises $21M to keep server clocks in sync by nightfly on Friday March 18, 2022 @10:52PM (#30730485)
statum-1 time server
Re:Clockwork raises $21M to keep server clocks in sync by galeaspablo on Friday March 18, 2022 @10:01PM (#30730180)
Yup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT.

The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious.

I’d love to see some papers, I went through all of Balaji Prabhakar‘s publications (titles only), and didn’t see a single paper that sounded like “NTP but machine learning.

If anyone else knows more about this, I’d love to hear from you. Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

Re:Clockwork raises $21M to keep server clocks in sync by kloading on Friday March 18, 2022 @10:54PM (#30730494)
I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article:

Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng

I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at explaining the general idea below.

Network edge-based measurement (one-way delay?): https://www.usenix.org/system/files/nsdi19-geng.pdf

Congestion Control: https://www.usenix.org/system/files/nsdi21-liu.pdf

Each paper I've listed builds on the last. Their method to synchronize clocks made accurate and efficient measurement of one-way delay possible with commodity hardware. This measurement of one-way delay occurs at the edge, allowing them to "hold" incoming packets at the edge for extremely small periods to reduce congestion (latency) while maintaining throughput. From my understanding, traditional congestion control algorithms require rich telemetry from the entire network, which is likely not accessible in a public cloud environment. Balaji and clockwork's algorithms only need to make these measurements from the edge (which customers in public cloud have access to).

I'm curious to see how all this will scale for multi-region deployments. If the latency between VMs from region 1 and region 2 is significant, I wonder if the measurement will actually be useful in deciding to "hold" the packets.

Re:Clockwork raises $21M to keep server clocks in sync by danzheng on Saturday March 19, 2022 @02:01PM (#30736017)
I'm from the Clockwork team, thanks for listing the relevant papers.

Accurate clocks sync enables true one-way delay measurements (instead of RTT/2), this allows for edge-based network visibility. We launched Latency Sensei beta – a sensor, monitor and auditor that provides visibility into cloud deployments. The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports include: 1)how VM colocation impairs network bandwidth, and 2) tale of 2 cloud regions, London vs Singapore. Take a look and we'd love to get some feedback https://sensei.clockwork.io/user/gallery/

On Congestion control, an edge-based solution is coming soon. If you're interested in a private beta, email us at [email protected]

Re:Clockwork raises $21M to keep server clocks in sync by galeaspablo on Saturday March 19, 2022 @05:07AM (#30732304)
This is exactly what I was looking for. Thank you. It’s a shame fundamental publications aren’t part of PR articles. I’d also love to see better details on their website. Oh well, I cant expect everything to work like academia — c’est la vie.
Re:Clockwork raises $21M to keep server clocks in sync by shiftingleft on Saturday March 19, 2022 @07:36AM (#30732979)
> I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release.

The first article you reference includes something about Support Vector Machines (SVMs).

Re:Clockwork raises $21M to keep server clocks in sync by insaneirish on Friday March 18, 2022 @11:19PM (#30730635)
> The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious.

They have the 'names' because they are a reputable group of people. They have been at this problem for a while, first with extensive research that required clock sync as a pre-requisite, and then to evolving into clock sync as a formidable problem unto itself.

Clockwork is a rename of the company as far as I can tell. Their original name (Tick Tock Networks) [1] was probably too close to what has become a very popular homophone.

[1]: https://opencorporates.com/companies/us_de/6947372

Re:Clockwork raises $21M to keep server clocks in sync by galeaspablo on Saturday March 19, 2022 @05:17AM (#30732347)
Sure, but ‘names’ don’t explain how they do what they claim to do. Another reply kindly provided references to publications (which I couldn’t find myself) — they made for great reading.

You seem to know their work, so if you have further publications, I’d love to get them please.

Re:Clockwork raises $21M to keep server clocks in sync by insaneirish on Saturday March 19, 2022 @06:25PM (#30738108)
Here's a talk Balaji Prabhakar gave: https://www.youtube.com/watch?v=YQSA99nrFEU. Looks like the clock sync part starts about 30 minutes in.

Haven't watched this one, but might be complementary: https://www.youtube.com/watch?v=Opf9CBwP5R8.

Re:Clockwork raises $21M to keep server clocks in sync by unmole on Saturday March 19, 2022 @12:44AM (#30731119)
> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

Juicero raised $120M.

Re:Clockwork raises $21M to keep server clocks in sync by charcircuit on Saturday March 19, 2022 @02:33AM (#30731683)
Juicero only raised $18.5M in their series A.
Re:Clockwork raises $21M to keep server clocks in sync by unmole on Saturday March 19, 2022 @02:35AM (#30731685)
And?
Re:Clockwork raises $21M to keep server clocks in sync by jacquesm on Saturday March 19, 2022 @05:34AM (#30732411)
> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

Just some words:

Theranos.

Ubeam.

Moller.

And many, many others. Investors do this all the time. FOMO + 'wouldn't it be nice' are powerful ways to part fools from their money.

Re:Clockwork raises $21M to keep server clocks in sync by isatty on Friday March 18, 2022 @10:55PM (#30730499)
> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

You’d be surprised, I was.

Re:Clockwork raises $21M to keep server clocks in sync by ahazred8ta on Saturday March 19, 2022 @12:50AM (#30731164)
From the article, time synchronization is only a small part of what they do. Their Big Thing is traffic shaping and latency management within and between datacenters.
Re:Clockwork raises $21M to keep server clocks in sync by galeaspablo on Saturday March 19, 2022 @04:59AM (#30732276)
Those products are built on time synchronization. E.g., a latency measurement needs a _true_ difference between the timestamps of two servers.
Re:Clockwork raises $21M to keep server clocks in sync by nikanj on Saturday March 19, 2022 @12:46AM (#30731132)
To be fair, their pitch deck did say both Machine Learning and Stanford
Re:Clockwork raises $21M to keep server clocks in sync by chillfox on Friday March 18, 2022 @10:13PM (#30730254)
Reminds me of early in my career where I encountered a home made time sync system because the creator had never heard of NTP.
Re:Clockwork raises $21M to keep server clocks in sync by stevekemp on Saturday March 19, 2022 @03:53AM (#30732030)
20 years ago I went on a ridealong with a bunch of men who were changing the times on the clocks installed in a bunch of church towers.

It was fascinating to get to watch the process. I'd always taken it for granted the big hotels, railway stations, and churches would often have big clocks you could see from a distance and that they'd be self-correcting. But of course a lot of old clocks are purely mechanical so when the time changed an hour forwards/backwards somebody would need to physically change them.

Re:Clockwork raises $21M to keep server clocks in sync by pmontra on Saturday March 19, 2022 @04:07AM (#30732080)
Even the clock in my car (2016) doesn't self correct. It goes ahead of about 1 minute every month.
Re:Clockwork raises $21M to keep server clocks in sync by jiveturkey on Saturday March 19, 2022 @12:35AM (#30731062)
How so? Clearly these guys have heard of NTP. They are talking about ns precision systems, not ms.
Re:Clockwork raises $21M to keep server clocks in sync by jrowley on Friday March 18, 2022 @10:39PM (#30730395)
Exactly! what would Leslie lamport say? ;)

https://lamport.azurewebsites.net/pubs/time-clocks.pdf

Re:Clockwork raises $21M to keep server clocks in sync by unfocussed_mike on Saturday March 19, 2022 @08:52AM (#30733354)
> it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea

Bless you. Are you younger than 40? :-)

Tech people who are over about 40 have, alas, no trouble imagining this at all!

Re:Clockwork raises $21M to keep server clocks in sync by oefrha on Saturday March 19, 2022 @12:43AM (#30731113)
Epitome of garbage HN drive-by dismissal: pick a seemingly stupid sentence from the article, quote it out of context, conclude that author/subject expert is clueless about <insert basic things>. Usually just a case study of the Dunning-Kruger effect.
Re:Clockwork raises $21M to keep server clocks in sync by evol262 on Saturday March 19, 2022 @08:17AM (#30733168)
Example of a garbage HN drive-by comment: this.

I've spent a _long time_ working in distributed systems with components which are "close to the metal". There are use cases for this, many of which are better solved by "hardware timestamping NICs" and "run a stratum 1/2 NTP server with high locality (and if you're geo-distributed, run multiple on systems dedicated to that purpose, because ensuring you're within nanoseconds of _someone else's_ infrastructure is generally an order or magnitude less important than internal coherence).

That was the best possible quote from the author/subject trying to explain the use cases. Did you even read TFA? It's completely unclear from their article what the intended uses cases are, what the problem space is, and how much working knowledge they have of existing solutions.

Naming people who invested in other companies who are investing in this one and showing what looks like a dashboard for a timekeeping solution is normal TechCrunch garbage, but that quote was a step above.

Comments like yours (going all the way back to /.) days really just tell me that you didn't read the article, and it makes it look like your primary goal is sophistry.

Re:Clockwork raises $21M to keep server clocks in sync by Nextgrid on Saturday March 19, 2022 @01:29AM (#30731369)
Given how much complete garbage has been VC-funded (or rather, overfunded, since the correct amount of funding should've been $0) it's good to remain skeptical.
Re:Clockwork raises $21M to keep server clocks in sync by oefrha on Saturday March 19, 2022 @02:09AM (#30731551)
If your way of remaining skeptical is opening the HN discussion and jumping into the first crappy hot take concluding that all people involved are clueless idiots without even skimming the article, your skepticism doesn’t mean much.