covi

Why do you target millisecond-level cycle time? Are you looking at very fine-grained tasks like elastic sandboxes, in addition to beefy pre- and post-training jobs?

This feels like the chimpanzee with a power drill. An agent is honestly just brute-force search, but guided.

The post says Slurm supports gang scheduling, k8s doesn't (out of the box).

Take a look at SkyPilot. Good for running these batch workloads. You can use spot instances to save costs.

To massively increase the reliability to get GPUs, you can use something like SkyPilot (https://github.com/skypilot-org/skypilot) to fall back across regions, clouds, or GPU choices. E.g.,

$ sky launch --gpus H100

will fall back across GCP regions, AWS, your clusters, etc. There are options to say try either H100 or H200 or A100 or <insert>.

Essentially the way you deal with it is to increase the infra search space.

Related: https://skyplane.org/en/latest/ (mentioned in OP)

From what I know this idea underpins a few FAANG-level companies' data transfer systems. OP's value = a simple implementation of the idea that's OSS and applied to AI.

Congrats on the API launch (from SkyPilot)!

If you want to use your own GPUs or cloud accounts but with a great dev experience, see SkyPilot.