Nevertheless, doesn't that force you to divide your work up into units somehow and manage external storage to track what/how much is done?

Yes, but this is common for "cloud native" applications. Web servers are ideally stateless "cattle", not "pets". Code is pulled down via CI/CD pipelines or via container images. State is in databases or blobs. Logs stream to some sort of analytics system.

Similarly, Kubernetes works well with spot VMs. You can have Spot and Regular node pools in one cluster, and then your workloads can happily spread out to the extra capacity at a huge discount.

It's really easy to "beat" the chaos monkey with this kind of design. You can destroy any VMs in our infra with reckless abandon and everything will be fine. If you destroy all the VMs of particular service obviously it will affect availability but it will still recover by itself in about 60 seconds.

You have to do that anyway for serious batch processing.

The reality at scale is that all cloud VMs are preemptible and unreliable. You can pay more to increase the probability of uptime but inevitably have to recover from failure sometimes.

The reality is itselt preemptible and unreliable, you could be terminated without notice

This is absolutely the correct way of thinking about this problem, and exactly how most people I've seen successfully use pre-empt instances treat them as well.

There are tools that enable this (e.g. look at temporal.io).

As for external storage: yes, you do generate more metadata, but that's often a tiny fraction of the cost saved by moving to spot machines.

In general, these tools are adopted more for increasing reliability of existing systems, but I predict they would be a neat fit to run them on spot machines.

+1 for Temporal. Using in prod, its great

(i work at Temporal) nice! glad you're happy with it, would love to learn more if you have any compliments/complaints, lets hear it :)

That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done.

I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.

You gotta do that anyway. It really sucks when a multi-day job crashes halfway through because of some silly bug and then you gotta restart from the beginning. The ability to resume from crashes is required regardless of where you're running the code.