> You think they don't checkpoint the model before feeding in a new slug of data?

They definitely do that otherwise they can't rewind the model when the loss shoots up. Unstable training happens to almost all LLMs, but can be managed by rewinding & skipping a few batches.