danzheng

Unbelievable. I can only imagine the product meeting where this feature was pitched and approved. Best wishes to Julia and the other writers.

We're scheduler agnostic and works well with autoscaling. We have tested against the k8s default scheduler and Volcano. Tests for Kueue and Yunikorn are coming shortly.

Hi all — I’m Dan, founding team at clockwork.io. Today we launched TorchPass! We'd love to get your feedback.

tl;dr: we built TorchPass because large distributed training jobs fail a lot, and checkpoint restart is expensive. TorchPass addresses this by migrating the state from failed resources to spares.

In large GPU clusters, even small failures (a GPU falling off the bus, a node crash, a network link flap) can bring down an entire distributed training job. And once you get into clusters with hundreds or thousands of GPUs, something is almost always failing. Research from Meta suggests mean time to failure drops to about 7.9 hours for a 1,024-GPU cluster. And when a single failure occurs, the entire distributed job crashes.

The usual recovery model is to take frequent checkpoints during training, and recover from the most recent checkpoint when a failure occurs. But:

all work since the last checkpoint is lost time is wasted replacing nodes and checkpoint reloading more time is lost restarting the entire distributed job

TorchPass uses a different approach: instead of restarting the job, it migrates the failed training rank to a spare GPU and resumes training at the same step.

TorchPass supports planned migration (triggered pre-emptively when an imminent failure is detected) or unplanned migration (triggered by a hard failure). Further details about how it works can be found here:https://clockwork.io/blog/torchpass-workload-fault-tolerance...

We ran a 3,000 step training benchmark using TorchTitan Llama-4 MoE Scout (109B) on 64 H200 GPUs with random failure injection to compare checkpoint restarts, TorchPass and TorchFT.

TorchPass completed in 405 min Checkpoint restart completed in 818 min. TorchFT in 930 min

Checkpoint restart was slower mainly because of the time taken to restore from checkpoint, restart the training and recompute the work since the last checkpoint.

TorchFT lost almost no time due to the failures, but was slower because it introduced a significant per-step overhead because it requires the using gloo (rather than NCCL) for cross replica all reduce operations.

Happy to answer questions about the implementation and benchmarks.

One thing missing in the blogpost is in practice you see many large orgs, especially in finance, living with multiple time domains. For example, on-prem trading systems almost always use PTP or PPS for sub-microsecond timestamping, often on dedicated networks to reduce jitter (for meeting regulatory requirements like MiFID II and CAT) while the rest of their infra (in on-prem and cloud) just runs NTP for millisecond-class sync. Both protocols are fundamentally sensitive to network conditions — the mean offset may look fine, but outliers due to congestion/jitter can be very poor.

The consequence of having multiple time domains is pretty painful when you need to reconcile logs or transaction histories across systems with different sync accuracy. Millisecond NTP logs and sub-microsecond PTP logs don’t line up cleanly, so correlating events end-to-end can become guesswork rather than deterministic ordering.

If you want reliable cross-system telemetry and audit trails, you'll need a single, high-accuracy time sync approach across your whole stack.

A very clever part of the HUYGENS algorithm is that it doesn’t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That’s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks.

The authors’ work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure

I'm from the Clockwork team, thanks for listing the relevant papers.

Accurate clocks sync enables true one-way delay measurements (instead of RTT/2), this allows for edge-based network visibility. We launched Latency Sensei beta – a sensor, monitor and auditor that provides visibility into cloud deployments. The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports include: 1)how VM colocation impairs network bandwidth, and 2) tale of 2 cloud regions, London vs Singapore. Take a look and we'd love to get some feedback https://sensei.clockwork.io/user/gallery/

On Congestion control, an edge-based solution is coming soon. If you're interested in a private beta, email us at [email protected]

Latency Sensei is build on top of accurate clock sync, it's a powerful sensor, monitor and auditor that provides visibility into your cloud deployment.

The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports are 1)how VM colocation impairs network bandwidth and 2) tale of 2 cloud regions, London vs Singapore, in terms of VM placement.

we use all the sensors on the device, including compass and accelerometer. This is only possible because we built our own end-to-end mapstack for mobile.

looks like the only cohort type you can select is acquisition date. mixpanel has more capabilities in this area.

curved display

someone in our shared office never connected her iPhone 3GS with her computer. She was still running 3.2.1 until I helped her to upgrade to 4.3.3.

San Francisco, CA -- Software Engineeer

EggCartel, www.eggcartel.com We're building a local mobile commerce platform to make buying and selling of used items as easy as taking picture on the phone. Office in downtown sf, here is the view from the office http://twitter.com/#!/danz/status/73505757181640704

We're looking for mobile engineers, backend engineers and generalists who can wear multiple hats. Please send me an email [email protected]