I believe that was asked for by the SREs- e.g. Tensorflow supports checkpointing to disk and restoring progress- but the ML training software used by the data scientists did not have this feature.